Video Description Markup Language

EXECUTIVE SUMMARY

I frequently watch tutorial videos on YouTube to hone my skills in several different practices.  Today I was watching a video and it occurred to me that videos would be much more useful if the elements in a video were independently addressable.  By that I mean you could copy and paste text from a video screen/scene, change your view or perspective on a scene, copy narrative into text, scale, rotate and translate objects from the video, including actors, furniture, spaceships, view a scene with and without the actors who were originally in it. 

In order to enable these activities and many more, it would be necessary for a standard(s) to be developed for creating and consuming this type of information.  Such a standard could be an XML derivative called Video Description Markup Language (VidDML).  VDML would be created to provide a datastream for consumers of this information and offers a new, novel, and superior way of transmitting video which provides for unique experience well above and beyond that available today.  Current video technologies and file formats such as MPEG-4 already provide for the ability to store and transmit such information (https://en.wikipedia.org/wiki/MPEG-4_Part_14), although not created specifically for working with VidDML.

The use of VidDML would provide a number of advantages over today’s video transmission.  Each individual media object in a video could be individually addressable, searchable, reusable, manipulatable and available for scrutiny.  Mashups could be created by consumers of VidDML / MPEG-4 streams to create new stories, facilitate understanding, enable search on images, characters, video, audio, scenes, emotions, speed, and automate a number of processes which are not possible with today’s current technologies.

DETAIL

I will further expound on how this process could work, using current technologies as an example to describe functionality, even though there is no current framework for the way this would work, and in order to enable these capabilities, new technologies or revisions of existing technologies would be needed to implement or enable these capabilities.

Creating Streams

To develop a data-stream, a VidDML / MPEG-4 source of data, many technologies need to be expanded, streamlined, and coordinated.  A good example of a video instance that uses multiple elements that would be all-encompassing is the evening news on television.  The evening news contains and is composed of people, music, speech, on-screen text, images, a variety of topics and specialties and expertise.

VidDML imagery could be composed using a device like Microsoft’s Kinect. The Kinect and its software had the ability to create 3D object point-clouds with colors and textures.

In our example of a news broadcast, a Kinect-like device would be used to create 3D “painted” models of the on-screen reporters, their desks and furniture, and objects in the reported stories such as crashed cars, guns, fire fighters and their vehicles, homes and business buildings and much more.

Not only should 3D models be created of the people and objects in the newscast, the actions of their facial expressions, limb and extremities, and emotional affect should be recorded and interpreted.

That covers the imagery aspect quite well, how about sound effects, music, and speech?

Sound effects could be imbedded in the VidDML / MPEG-4 stream as well.  The sound effect itself could be stored in a library and referenced, or directly provided in the stream.  Music would be similar.  A riff or the audio of a musical score could be stored in the stream.  Better yet, the musical score representation would be embedded in the stream and would be searchable and independently addressable.  Perhaps the best idea would be to leverage MIDI technology and embed MIDI commands in the stream.

Speech is similar.  There should be a textual representation of the spoke word.  Voice information could be embedded also, so that a text-to-speech capability could use phonemes of an actual individual to provide searchable speech.  Data that modifies inflection, volume, and emotion should be streamed as well.  Text-to-speech technology needs finessing, like the other technologies mentioned in this document, it is not quite ready for prime-time for use in our example of news castors. 

Consuming Streams

To implement a VidDML / MPEG-4 based solution, an enormous amount of computing power is needed.  Today’s 3D video games stretch the boundaries of what is possible, and do not facilitate the manipulation of individual media elements in real time.  Video games also do not provide real-time search and rule-based interpretation/manipulation along all the media channels.

2D / 3D Objects - Orientation / Perspective

As mentioned earlier, a device like Microsoft’s discontinued Kinect can create 3D objects with color and textures.  Other companies have technologies which provide the same result.  In our news presentation, the news castors could be imaged in 3D, and the viewer of the newscast could place themselves in various locations in the studio. In addition, news stories on location could use 3D imaging devices to create 3D “scenes” where the viewer could be placed within the scene and view in varying directions. 

Another possibility, when technology enables it, would be to provide for the manipulation of 3D models of news castors, news story subjects, and actors.  There could be an action stream in the VidDML / MPEG-4 that specifies the visible activities of the newscastors.  There could also be an emotion stream that enables modifications to the voice and actions of actors that modifies and acts as a filter to impact what the viewer sees. 

Having 3D models of the newscastors, an emotion stream and a physical action stream would enable the viewer to substitute the actor of their choice as a newscaster, complete with their gestures, mannerisms, and voice for that provided by default by the producer of the newscast.

Sound / Speech / Score / Music

Sounds could have multiple channels in the stream.  Descriptions of the sound, its volume, and other audio characteristics could be projected, searched against, and modified.  Similarly, speech could have a channel where the speakers voice, emotional affect, and vocal mannerisms are represented.  Music streams could be composed of MIDI data, modifiable, searchable, and replaceable.

Actors / Activity / Emotion

Each actor / newscaster would have associated physical actions, voice, mood, emotional affect, all of which would be encoded, stored, searchable, and modifiable.  These characters are not only representative of people, could be animals, machines, indoor and outdoor objects.

Text

A text stream is the most useful to implement.  This stream would be encoded into the VidDML / MPEG-4 data.  Every other stream would have a textual representation to facilitate searching, modifying, and reuse.  If there is text in the video, like the other types of media, you would be able to select and copy/paste from a video into other apps.  You could also copy from other apps and place modified text to modify other streams.

Close

Herein is a description of a capability which consists of many parts.  Many of the technologies mentioned exist in one or another states of maturity.  I expect a usable system using what is discussed could be created with an acceptable level of finesse in 5-10 years. 

Equipment, software, and usability need to be developed to enable the possibilities described herein.  Sounds like fun!  I would like to help make this happen!

Virtual to Real Workflow with Future Visio Tools

In the near future, I expect that tools such as Microsoft’s Visio (Microsoft, 2017a) will be able to create workflows of automated equipment.  Not just virtual representations of workflow, but actually send instructions to knowledge-worker software and manufacturing systems to configure them on the fly and immediately start creating product.  

I envision this working with 3D printer farms, robots, and automated finishing equipment, all of which could be mobile on a factory floor such that a workflow could be created in Visio or a Visio-like tool, hit a “Create” button, and the equipment on the factory floor moves itself into position for an optimal workflow with robots, 3D printers, inventory, and finishing equipment to facilitate the production of both small and large quantity manufacturing.  

Workflow would not be limited to manufacturing equipment, but could apply to knowledge worker processes as well.  Nodes for engineering work, graphic design, 3D simulations, marketing, fund-raising and other functionality could be tied in.

References

Microsoft. (2017a). Visio Pro for Office 365. Retrieved from https://products.office.com/en-us/Visio/microsoft-visio-pro-for-office-365?wt.srch=1&wt.mc_id=AID623587_SEM_4xYGC7RC

Pearlson, K. E., Saunders, C. S., & Galletta, D. F. (2016). Managing and using information systems: A strategic approach (6th ed.). Hoboken, NJ: John Wiley & Sons

Consider the Origination of Thought and 3D

Consider the origination of thought, expression of design on a PC, tablet, or mobile device, or by capturing a picture drawn on a surface and sending of an image of an object design to India, China, or Eastern Block countries for conversion into a computer aided drafting (CAD) file (Kim, 2015), which is then sent back to the originator, and then forwarded onto an individual or company that does 3D printing from CAD (usually a .STL file).  

The characteristics of the company doing 3D printing should be considered, specifically quality and complexity, printing fees, shipping fees, and interaction with the customer such that if something in the CAD file looks amiss, they will alert the customer as to the issue, and possibly suggest corrections to the file, or the 3D printer company staff may be able to correct issues with the CAD file.  In addition to quality issues, the location of the 3D printer is to be considered as well as the quality issues as the distance shipped, and the weight of the 3D printed object affect the cost of the entire process.  

References

Baltzan, P. (2015). M: Information systems (3rd ed.). New York, NY: McGraw-Hill.

Kim, S. (2015). 3D model data generation and conversion for 3D printers. Journal of Electronic Materials, 44(3), 815-822. doi:http://dx.doi.org/10.1007/s11664-014-3584-5

3D and the Time Machine

In chapter five, Sousa and Oz (2015) discuss 3D Geographic software such as Google Maps and Trimble's SketchUp (formerly owned by Google) and suggests uses for this type of software by city service providers such as fire departments such that the fire departments can locate fire hydrants in an emergency.

The use of 3D is becoming more prevalent, especially with the development and distribution of devices such as the Microsoft Kinect.  Devices such as the Microsoft Kinect incorporate "a depth sensor, a color camera, and a four-microphone array that provide full-body 3D motion capture, facial recognition, and voice recognition capabilities" (Zhang, 2012, p. 4). 

If one extends the technology into the future, it would appear that such devices would be incorporated on a global scale, in a time independent way.  What this means is that both outdoor and indoor activities would be recorded and made available to those authorized to view or work with such content.  An example would be that an individual in a home could be recorded from the time they woke up in the morning, went through their morning routine, recorded getting into their car, recorded driving to get their morning cappuccino, then to work, and all of their foot traffic there would be recorded, then their travel to little Jimmy's baseball game where all of the players and play would be recorded, then travel back home, and the evenings activities in the home would be recorded, then to bed, and the next day would start anew.

The interesting aspects of this type of recording is that as the data recorded includes both imagery, sound, as well as 3D coordinates, a viewer could place him or herself anywhere in the mix and get a perspective from that location live at any time in recorded history.  Mix in a bit of hypermedia and you (with permission and authorization) could visit an accident scene, review the accident from several angles, click on a car, then click on a person in the car, and get the individuals medical records, from previously broken bones, to dental films, to current medications.  It would essentially be a time machine on steroids with a "view only" capability.  An individual would be able to "rewind" an accident scene to the point at which one of the cars in the accident was seen stopping at the local pub to "chug a few" with the guys. 

This could revolutionize society as we know it.

References

Sousa, K.J., & Oz, E. (2015). Management information systems (7th ed.). Stamford, CT: Cengage Learning.

Zhang, Z. (2012, April). Microsoft Kinect Sensor and Its Effect. IEEE MultiMedia, 19(2), 4-12. Retrieved from https://www.microsoft.com/en-us/research/publication/microsoft-kinect-sensor-and-its-effect/

A Revolution in Payment Systems that is Occurring Even Today.

One element in the Sousa and Oz (2015) chapter 4 caught my attention.  They discussed a credit card processing system called Square, which is one that my computer tech uses when I pay him.  This is part of a revolution in payment systems that is occurring even today.  There exist two sides to payment processing, making a payment and accepting a payment.  Apple (Apple, n.d.) has created a system called Apple Pay whereby you can use your Apple device to make payments wirelessly by placing your Apple device next to a complementary payment device.

Chase Bank (Chase, n.d.) also has a payment system.  Chase covers both sides of the payment equation with the ability to both make payments and accept payments using their app on your mobile device.

These three systems provide benefits for both Fortune 500 companies as well as the small office/home office (SOHO) worker.  Now the independent knowledge worker is on an even par with larger firms in terms of making and accepting payments for products or services rendered.

References

Apple. (n.d.). Set up Apple Pay on your iPhone, iPad, Apple Watch, or Mac. Retrieved from https://support.apple.com/en-us/HT204506

Chase. (n.d.). Chase Mobile® Banking. Retrieved from https://www.chase.com/online/digital/mobile-banking.html

Sousa, K.J., & Oz, E. (2015). Management information systems (7th ed.). Stamford, CT: Cengage Learning.

One of the most incredible things I can think of using 3D for is the design, visualization, and marketing of products that have not yet been created!

One of the most incredible things I can think of using 3D for is the design, visualization, and marketing of products that have not yet been created!  For instance, if I have an idea for a cool widget, but I don't have the cold hard cash to create one, I could have the widget modeled in 3D, create an advertisement showing the widget in use by consumers in a video using the widget's 3D model, and offer the widget for sale on a website with various views and perspectives, and evaluate the demand of the widget based on website sales, calls, and various other inquiries.  

For those early adopters, I could 3D print the cool widget, and charge a premium, and if demand warranted, I could use the 3D model of the widget to create castings, molds and the tool and die necessary for mass production.  The interesting aspect of having a virtual item is that you can present it visually for promotional purposes, without the time, effort, and expense of actually creating a real widget, and save the expense in time and money until you have actually sold the widget!  Then 3D print or manufacture based on demand.

The blue doodad in the picture is a webcam mount I designed and had 3D printed for my robot after I had taken a CAD class at the local community college to prepare myself for writing a dissertation on something related to 3D printing.  

Data Analysis for Small Firms / SOHO

In chapter 11 Sousa and Oz (2015) discuss the various ways of processing data and gleaning meaning, trends, and patterns within the data.  These methods go by the names of Business Intelligence (BI), Online Analytical Processing (OLAP) Data Warehousing, and Data Marts (Sousa & Oz, 2015), I have a bit of experience in Microsoft Access and SQL Server databases, and although what I have done may not be as sophisticated as some of the applications in chapter 11, they have served their purpose.

For instance, a friend of mine wanted to send out a postal mailing for a special offer.  He wanted the offer to be available to his best previous customers in a four-state area in order to minimize shipping costs.  He has data available from his online store.  So we downloaded the data, and imported it into Microsoft Access.  In slicing and dicing the data, it became apparent that the data was not "clean."  In other words, it was not normalized.  By sorting on the criteria of last name, first name, ZIP code it became readily apparent that some of those who we thought were "good" customers were actually "great" customers.  Other customers, those who appeared in the data to have just purchased small items here and there were actually "great" customers as well, they made small purchases, yet they made several purchases which added up to a substantial sum over the course of a few years.  This is an example of how even small companies can mine their data to realize a benefit with desktop database tools. 

Reference

Sousa, K.J., & Oz, E. (2015). Management information systems (7th ed.). Stamford, CT: Cengage Learning.

What trends do you predict for the future of organizational theory and design?

Dr. Consuelo asked:

"What trends do you predict for the future of organizational theory and design? What organizational design challenges could the future hold" (Meux, 2014, para. 1)?

I believe that automated management tools, process graphing tools, and flexible, programmable manufacturing and service providing devices will come together to provide for a rapid design, rapid change, and rapid configuration of business processes with a minimal headcount.

This is the vision. Say that Bob has a desire to sell a widget that he just thought of. He will be able to have it designed with globally available low-cost talent using a simple online 3D modeling and design package. He will be able to connect with a business service provider for manufacturing, logistics, and marketing services. Possibly each from a different vendor. 

After the creation of the 3D design for his product, Bob will use a Microsoft Visio-like program to draw and create a process for taking in inputs such as the 3D design, raw materials for the rapid-prototyping machine/3D printer, boxes and packaging material. Bob will then tie together the packaging process, marketing, and logistics, again through a process design drawing tool and helpful wizard-based interface. Each part of the process will be defined, drawn, and submitted to a service bureau. 

The drawing/design is more than a drawing. Through computer technology and Internet access, the drawing/design drives and configures the equipment, processes and the people involved in the processes. Once Bob has submitted the process design to the service bureau, and paid for it with funding, also acquired through an online funding acquisition tool and wizard-interface the widgets will be created, marketed, sold, and delivered. Not only that, but he has been able to do all this while wearing his PJs and in a 24 hour period.

In terms of how this will impact organizations, organizations will become much smaller, more flexible, a substantial reduction in headcount. In addition, many services and products will be able to be designed, created and shipped at and from a quality, low-cost global provider of products and services. Automated online systems will automatically connect, deliver, and produce from low-cost, global providers, all driven automatically by a drawing/design produced by MS-Visio or a similar tool.

Future challenges: There will be a substantial reduction in the headcount needed for many service and manufacturing businesses. Unemployment will be severe. The reduction in headcount will be taxing for our nation. On the flip-side, it will be easier and faster for people to create new businesses with just a simple idea in mind. The challenge is this: Can individual’s creativity ramp-up employment more than the reduction taking place in headcount for service and manufacturing based businesses in our nation? 

References

Meux, C. (2014, Aug.). University of Phoenix - New Classroom. Week 8 Participation Posting.
Richards, A. (2014, Aug.). University of Phoenix - New Classroom. Week 8 Participation Posting.
© 2014 - Aaron L. Richards. All Rights Reserved. All names, trademarks, and content owned by their respective entities.

Above and Beyond the Five Senses You Learned in School

It has been a while since I have blogged. Here is something I have rolling around inside my head, trying to get out. I worked on it while walking the mall which I do for exercise on a near daily basis.

In elementary school / middle school when studying the science of humanity you learn about the five senses, 1) Taste, 2) Touch, 3) Smell, 4) Hearing, and 5) Sight.

However as we age, we learn about the additional senses the human body is capable of. Many are casually recognized as senses, having the word "sense" in their name, even though they are not formally or scientifically recognized as such. Here are the ones I was able to come up with, I am sure there are many more.

1) Sense of Direction. 2) Sense of Time. 3)  Sense of Being Observed. 4) Sense of Other Person's Emotional State. 5) Sense of Belonging. 6) Sense of Fashion. 7) Sense of Style. 8) Not sure that this is the same thing, but Common Sense.

Most of these are self-explanatory. I am curious as to what you can come up with in terms of senses over and above the 5 senses we learned in school.

Time and Space Machine

Creating a Time and Space Machine with Today’s Technology

(C) 2005 Aaron L. Richards

A Time and Space Machine - Able to traverse time and space at a moments notice. Just select a location, and a point in time and ba-boom, you are ready to fight evil aliens, and others bent on doing harm to our world. It is a premise filled with all sorts of contradictions as to altering the time-space continuum.

Popular fiction? An advanced future technology? I don't think so. We can start recording time and space for use in our Time And Space Machine today. We can do this by extending currently available services through the use of popular and commonly available technologies. We can make the Time and Space Machine available through the World Wide Web. What we are in need of is the vision. Here it is!

Concept of a Time and Space Machine in the Popular Media

According to popular fiction, a Time And Space Machine is a device that bodily places one or more individuals over geographic space and through time to different location and a different time in the past or future. Once there, an individual can often view (and interfere with) great moments in history, and change the present in an unexpected way. At other times, individuals go into the future, and bring back devices of great or terrible purpose, which have been developed in our future and which have capabilities for which we are unprepared to accommodate, and which cause problems in the current time.

How We Are Going To Define A Time And Space Machine

I would like to define a Time And Space Machine as a service that allows individuals to see images of different locations at different points in time in the past. It would only be able to view such images of locations at times that had previously been recorded. The viewing of such recorded "history" would occur on a personal computer or workstation. I see such a service being made available on the World Wide Web. I will describe what the Time And Space Recorder consists of later on in this article. A Time And Space Recorder will be designed to be used on the worlds roadways as I envision it, and as a result the Time And Space Machine will primarily be designed to view scenes from automobile traveled roadways.

Such a system would have the ability to go forward or reverse in time while focusing on a view of a specific location provided recorded data existed for that location.

Similarly, geographic space could be traversed, while time was frozen.

In addition, one could traverse both time and space at the same time. Either in forward or reverse order, at "normal" speed, or fast forward or fast reverse, kind of like watching a DVD of a travel video.

What Would a Time And Space Machine Consist of in the Way of Equipment?

There would be a Time And Space Viewer, a Time And Space Server and there would be the data acquisition component which earlier I referred to as the Time And Space Recorder.

The Time And Space Viewer would be a personal computer or workstation with a web-browser program. All that it needs to do is view and interact with the Time And Space Server as a standard PC interacts with a web server.

The Time And Space Recorder would consist of a Global Positioning Satellite receiver, one or more video cameras, a video digitizer if the video cameras were not digital, a cellular broadband modem, a small computer, an internet connection, a database program, and a computer program that could tie it all together.

This equipment would be tied together by the computer in a system that allowed the video camera(s) to take in video as a vehicle drives down a road, encodes each frame of video with a latitude, longitude and orientation (direction) taken from the Global Positioning Satellite receiver as well as a time stamp. It could then transmit these images with their associated data to the Time And Space Server. This data would be transmitted through the cellular broadband modem.

The Time And Space Server would be part web-server, part specialized search engine. It would match time and space queries received from the Time And Space Viewers to the data supplied it by the Time And Space Recorders.

How Would This Time And Space Machine Work? What Would It Do?

In order to be feasible, this system would have to be deployed something like an On-Star system with video capability. Cars and trucks would be equipped with Time And Space Recorders as described above. As vehicles traverse the worlds roadways, they would capture video all along their routes. Each frame of video would be encoded with time and location data and sent to the Time And Space Server. If the Time And Space Recorders became commonplace, then many popular roadways would be traveled by these systems several times an hour.

Traverse space

If you had a large number of vehicles that were equipped with the Time And Space Recorders, that were traveling over a given geographic space, collecting images at roughly 30 frames/second, and each frame of video was being encoded with latitude and longitude from the GPS system, there would be quite a collection of images that would span a large geographic space.

If the appropriate software systems were in place to locate and traverse this information (the Time And Space Server), allowing the aggregation of all of the information from all of the vehicles with TASRs that were on the road, one could have a system such as that in the following description:

One could have a website that was similar to today’s map websites such as MapQuest, Yahoo Maps or Google Maps. The individual could type in a specific address or intersection and in return receive an image of a map. Like Google Maps, this map would be scrollable by clicking and dragging the mouse. In addition, there would be an image from a Time And Space Recorder (TASR) on the web page of the given location. If you moved the mouse, guiding a cross-hair around the streets on the map, images that had been acquired by TASRs at that geographic point could be displayed real-time (with a broadband connection) on the web page with the map. Not all of these images would necessarily come from the same vehicles TASR. The images displayed could potentially be from tens or even hundreds of vehicles systems aggregated over time into a system that coordinated with a mapping system on the Internet.

Traverse Time

Now that we can traverse space, how do we traverse time?

Well, taking the system described above, with images aggregated over a large geographic area from multiple vehicles' TASRs, note this: in areas with more traffic, it is highly likely that certain areas will be traversed by TASRs multiple times a day, perhaps, depending on how prevalent these systems are, maybe multiple times an hour. Now in these places that are traversed by TASRs multiple times an hour, if this data is recorded, there will be multiple images of a location per hour, for as long as the information is recorded, which at some future time could potentially be years worth of images. Now imagine the same mapping website as mentioned above. A user puts an address or location in the mapping website, and up pops a map of the surrounding area with a star at the specified location, as well as an image as viewed from the TASR of a vehicle at that location. Now in this scenario, imagine that this page would also have a clock and calendar on it. As you move the hands of the analog clock on the page, images taken of that location closest to the time specified would be displayed real time on the web page. If you wanted images from a different date, you could change that calendar date and the image of that location, closest to the date and time specified would be displayed. It is highly likely that the images would not all be from the same vehicle or TASR, but from an aggregated pool of images coded for that geographic location.

Now Imagine – Time and Space Traversal

Just imagine what would be possible if the number and density of the TASRs was great enough. It would be possible to view the scene of an accident, such as a two car collision using this system, and then follow either vehicle in reverse through time and space to see the events and behavior of the vehicles before the accident. You could, provided the image data resolution was great enough actually watch the accident occur in the form of a video clip. There are a number of possibilities, some very beneficial, and others more reminiscent of the Surveillance Society of Big Brother to be feared.

Although such a system as described may raise certain fears, being bodily transported into the past to battle aliens, and in the mean time disrupting the time-space continuum is highly unlikely to happen given this technology. Or is it?

BrainStormer II

I've decided not to develop and submit my software to Momentum. Nothing wrong with Momentum, I had run into a set of technical issues which I debated on the likelihood of being able to overcome (and I have since overcome.)

Since last blog article I have created a set of wireframes and storyboards and begun the process of database design. I am using SQL Server Express 2008 to create the database, unfortunately the hosting service I use is SQL Server 2000, so there is a bit of extra work involved in porting the structures over. I'll have to see if the hosting service will support the Express 2008 files.

I've also created a couple of pages and connected them to the database using Visual Studio 2008.

I'll keep you updated as I progress.