Have a long time interest in code breaking and its connection to the birth of the modern computer. Much of that happened at Bletchley Park. Here a text and image tour with no technical detail. Some of the earliest examples of human assisted machine learning patterns in complex data. I spent some time working in the rooms at the University of Pennsylvania that had held the early ENIAC computer in 1943. But that was no where near as dramatic as Bletchley Park.
Saturday, May 31, 2014
A Tour of Bletchley Park
Have a long time interest in code breaking and its connection to the birth of the modern computer. Much of that happened at Bletchley Park. Here a text and image tour with no technical detail. Some of the earliest examples of human assisted machine learning patterns in complex data. I spent some time working in the rooms at the University of Pennsylvania that had held the early ENIAC computer in 1943. But that was no where near as dramatic as Bletchley Park.
App for the Physically Impaired
On building apps for the physically impaired. From Phys Org News. " .... A new smartphone app developed by a team at the University of Palermo in Italy helps sightless people navigate inside of buildings where GPS doesn't work. The development effort was paid for by the Andrea Bocelli Foundation and the result is meant as a means of aiding the blind in finding their way around inside of buildings by following a pre-designated path. The app works in conjunction with special tape that is placed on the floor. The user points the smartphone at the floor in a way reminiscent of a cane, swiping it back and forth until it buzzes in their hand, letting them know when to go straight or turn. ... " Site includes a link to a preprint paper.
Scratch Creation Language from MIT
Scratch. Was pointed out to me a year ago but just had a chance to take a look. Has existed publicly since 2006. Creation environment for kids. This reminds me of other environments, like Turtle Programming of longer ago. This adds sharing and takes the idea of creating more broadly. " ... Create stories, games, and animations ... Share with others around the world .. ." . Also the idea of providing help through a process of curation.
" ... With Scratch, you can program your own interactive stories, games, and animations — and share your creations with others in the online community.
Scratch helps young people learn to think creatively, reason systematically, and work collaboratively — essential skills for life in the 21st century.
Scratch is a project of the Lifelong Kindergarten Group at the MIT Media Lab. It is provided free of charge. ... "
" ... With Scratch, you can program your own interactive stories, games, and animations — and share your creations with others in the online community.
Scratch helps young people learn to think creatively, reason systematically, and work collaboratively — essential skills for life in the 21st century.
Scratch is a project of the Lifelong Kindergarten Group at the MIT Media Lab. It is provided free of charge. ... "
User Generated Content and Media Perception
In Knowledge@Wharton: The addition of often politically polarized user-generated content changes public perception of media organizations – and it may hurt profits. But competitive pressure means it is likely here to stay, says Wharton’s Pinar Yildirim. ... "
SEO at Tesco
An example of a technologically advanced retailer:
" ... Search engine optimization has helped Tesco grow to occupy 50% of the U.K. online grocery market, and among 2,500 online search terms, Tesco is the most visible supermarket by a significant margin, according to data from Pi Datametrics. Tesco's competitors, which include ASDA, Waitrose and Sainsbury's, have made strides in the past year to boost their SEO, but their average positions, while visible in a search, are poor and correspond to a small number of search terms, Intelligent Positioning Director Sam Silverwood-Cope writes. ... "
" ... Search engine optimization has helped Tesco grow to occupy 50% of the U.K. online grocery market, and among 2,500 online search terms, Tesco is the most visible supermarket by a significant margin, according to data from Pi Datametrics. Tesco's competitors, which include ASDA, Waitrose and Sainsbury's, have made strides in the past year to boost their SEO, but their average positions, while visible in a search, are poor and correspond to a small number of search terms, Intelligent Positioning Director Sam Silverwood-Cope writes. ... "
Buzzradar: Social Media Captured, Visualized
Just brought to my attention: Buzzradar Social Media: Captured, Visualized. " ... We monitor real-time social media content for your brand or event and visualise it in engaging, meaningful and entertaining ways. ... " Also @Buzzradar. See also a video here. And their blog. More to follow.
Friday, May 30, 2014
Predicting World Cup Outcome with Bayesian Networks
A nice example of how a Bayesian network can be used to do predictive analytics. Much more here. " ... Designed around a Causal Bayesian Network, and using the BayesiaLab's API, the BWCP computes each team's qualification probabilities for proceeding from the Group Stage to the Knockout Stage in the 2014 FIFA World Cup. ... ". At the link you can give it a try.
Apache Spark for Hadoop
Had not seen it before, but was recently shown an early implementation of Spark and how it could be used to efficiently drive Hadoop. Impressed by what I saw. It appeared that it could be used for both straightforward and relatively complex applications. Recently in the news: Apache has released the first production ready version of Spark.
DARPA Talks Innovation Direction and Marketing
A nice piece on a recent talk by DARPA director of innovation, Dan Kaufman. Have had a long time connection with them, from early Pentagon days, to fairly recently. Well, then we all have been using their developments for some time now. So whats next? Looks like a push in the re-emergence of AI. More embedding and hacking. He makes it clear they are deep level innovators, but not marketers. Do keep it up, we can market for you. Spend our money wisely. Link to the talk is included.
Self Study Machine Learning
Jason Brownlee writes about machine learning, worth a look. For the purposes of this work, and most others machine learning is the term used by computer scientists for data mining. The latter is more often used in business.
" ... You need a structured approach.
A framework for self-studying machine learning will focus your time on very specific outcomes. A structured approach can rapidly accelerate what you can achieve with machine learning.
I have created a self-study guide designed for programmers, engineers and students comprised of 4 competency levels in a 26-page PDF guide called: The Self-Study Guide to Machine Learning
If you want a structured approach to self-studying machine learning, then this guide is for you! ... "
" ... You need a structured approach.
A framework for self-studying machine learning will focus your time on very specific outcomes. A structured approach can rapidly accelerate what you can achieve with machine learning.
I have created a self-study guide designed for programmers, engineers and students comprised of 4 competency levels in a 26-page PDF guide called: The Self-Study Guide to Machine Learning
If you want a structured approach to self-studying machine learning, then this guide is for you! ... "
Estonia as E-Society
In the CACM: A survey of the use of electronic systems for commercial, social and government interactions in Estonia. We can learn from small country experiments with technical adoption.
Social Media Influencing Companies
This question came up recently. Do we have rough numbers about how much social media influence companies? Of course it depends upon how you define influence and on what topic. SmartKPIs takes a cut at this.
Cognitive Miser Theory
The Cognitive Miser, applications to consumer engagement?
:
" ... an umbrella theory of social cognition that brings together previous research on heuristics and attributional biases. The theory suggests that humans, valuing their mental processing resources, find different ways to save time and effort when negotiating the social world. The term cognitive miser was first introduced by Susan Fiske and Shelley Taylor in 1991. ... "
:
" ... an umbrella theory of social cognition that brings together previous research on heuristics and attributional biases. The theory suggests that humans, valuing their mental processing resources, find different ways to save time and effort when negotiating the social world. The term cognitive miser was first introduced by Susan Fiske and Shelley Taylor in 1991. ... "
Thursday, May 29, 2014
Intel in Your Autonomous Car
Intel chips for automobiles. A big market, so the reach is obvious. Specifically the mention of autonomous is interesting. What are the specific needs of such a chip. Speed and reliability? Embedded intelligence and connectivity.
" ... Intel threw its research might behind the idea of autonomous cars, or automobiles that are safer and more efficient because they can take over the task of driving from humans.
Intel, the world’s biggest chip maker, will provide an “application ready platform” with its own processors and operating system for self-driving cars. The initiative is part of Intel’s larger campaign to provide the intelligence and connectivity for the internet of things, or smarter everyday devices. The effort will include Intel’s own research as well as investments in other companies from a $100 million car technology fund. .... "
" ... Intel threw its research might behind the idea of autonomous cars, or automobiles that are safer and more efficient because they can take over the task of driving from humans.
Intel, the world’s biggest chip maker, will provide an “application ready platform” with its own processors and operating system for self-driving cars. The initiative is part of Intel’s larger campaign to provide the intelligence and connectivity for the internet of things, or smarter everyday devices. The effort will include Intel’s own research as well as investments in other companies from a $100 million car technology fund. .... "
Trucks Tethered by Computer Control
In Technology Review: Would have fit right into some of our fleet simulation work. This has the potential for saving fuel. But also appearing frightening on the highway. Regulation exists by US state and probably has not been altered for this yet. " ... 10-4, Good Computer: Automated System Lets Trucks Convoy as One .... A recent demonstration involving two trucks tethered by computer control shows how automation and vehicle-to-vehicle communication are creeping onto the roads. ... "
IARPA Sharp Meets Big Data
I was reminded of the IARPA Sharp project again. To which I have a loose connection. Some of the statements below are certainly how we define intelligence ... being adaptive under varying and often contradictory data. This means something different again under the strain of big data. Big Data means we need analytical methods as part of our understanding of the data itself. More thoughts on this will follow.
Strengthening Human Adaptive Reasoning and Problem-Solving (SHARP)
Adaptive reasoning and problem-solving are increasingly valuable for information-oriented workplaces, where inferences from sparse, voluminous, or conflicting data must be drawn, validated, and communicated—often under stressful, time-sensitive conditions. In such contexts, one’s ability to accurately update one’s mental models, make valid conclusions, and effectively deploy attention and other cognitive resources is critical. .... "
Strengthening Human Adaptive Reasoning and Problem-Solving (SHARP)
Adaptive reasoning and problem-solving are increasingly valuable for information-oriented workplaces, where inferences from sparse, voluminous, or conflicting data must be drawn, validated, and communicated—often under stressful, time-sensitive conditions. In such contexts, one’s ability to accurately update one’s mental models, make valid conclusions, and effectively deploy attention and other cognitive resources is critical. .... "
Visualization Making Big Data Meaningful
In CACM: Includes some videos of data artists at work.
" ... Until recently, a good spreadsheet or perhaps a pie chart might have been sufficient to get a firm grip on a dataset. However, making sense of big data requires more, and with our increasing inundation with data comes new and creative opportunities to build unique interfaces.
That is why, powered by new technologies such as touchscreens and giant LCD monitors, so-called "data artists" are experimenting with ways to make huge amounts of data comprehensible and accessible to a broader range of people, and then displaying their data visualizations inside venues like the corporate facilities of companies like Microsoft and Google. ... "
Note also the leveraging of multiple display mediums, like that experimented with using Business Sphere. Big and complex data can bring patterns to the surface and let our eyes do the pattern recognition.
" ... Until recently, a good spreadsheet or perhaps a pie chart might have been sufficient to get a firm grip on a dataset. However, making sense of big data requires more, and with our increasing inundation with data comes new and creative opportunities to build unique interfaces.
That is why, powered by new technologies such as touchscreens and giant LCD monitors, so-called "data artists" are experimenting with ways to make huge amounts of data comprehensible and accessible to a broader range of people, and then displaying their data visualizations inside venues like the corporate facilities of companies like Microsoft and Google. ... "
Note also the leveraging of multiple display mediums, like that experimented with using Business Sphere. Big and complex data can bring patterns to the surface and let our eyes do the pattern recognition.
Algorithms That Dominate the World
A fun example list of how algorithms are being used today in just about everything we do. At first I thought these were detail level examples, but they are more process level examples. For example the process of Search, Adwords, Dating matches, Trading, Data gathering. Also, there is not much detail here. Still worth a look to understand some of the basic yet most important algorithmic engines operating today.
" ... The importance of algorithms in our lives today cannot be overstated. They are used virtually everywhere, from financial institutions to dating sites. But some algorithms shape and control our world more than others — and these ten are the most significant.
Just a quick refresher before we get started. Though there's no formal definition, computer scientists describe algorithms as a set of rules that define a sequence of operations. They're a series of instructions that tell a computer how it's supposed to solve a problem or achieve a certain goal. A good way to think of algorithms is by visualizing a flowchart. .... "
" ... The importance of algorithms in our lives today cannot be overstated. They are used virtually everywhere, from financial institutions to dating sites. But some algorithms shape and control our world more than others — and these ten are the most significant.
Just a quick refresher before we get started. Though there's no formal definition, computer scientists describe algorithms as a set of rules that define a sequence of operations. They're a series of instructions that tell a computer how it's supposed to solve a problem or achieve a certain goal. A good way to think of algorithms is by visualizing a flowchart. .... "
Wednesday, May 28, 2014
Consumer Goods Visionaries: P&G's 1- Consumer Place
I see that Steve Aaron, former colleague, has been identified in Consumer Goods Technology as the architect of P&G's revolutionary 1, Consumer Place. See an interview with him here. A number of interesting levels of details are included. Congratulations, Aaron!
" ... he architected 1, Consumer Place, a figurative street address designed to consolidate all of P&G’s marketing programs into a single, secure and scalable global marketing platform and ecosystem, enabling coordinated, analytics-driven decision making across the entire company. For the first time, P&G is now able to see what is working and share solutions and best practices across brands. ... "
The post also contains a number of practitioners in other consumer goods companies.
" ... he architected 1, Consumer Place, a figurative street address designed to consolidate all of P&G’s marketing programs into a single, secure and scalable global marketing platform and ecosystem, enabling coordinated, analytics-driven decision making across the entire company. For the first time, P&G is now able to see what is working and share solutions and best practices across brands. ... "
The post also contains a number of practitioners in other consumer goods companies.
Content Aggregation by Retailers
In Retailwire: Interesting thought, but also check out the comments at the link, which points out some problems. The piece points to this article. ' ... What do you think of the prospects of a central shopping/loyalty site supporting a community of retailers? ... ' " ... Alternatives are emerging. Blogger-fed websites and the more established coupon distribution sites are beginning to build a following with shoppers who seek deals and content as they plan their next shopping trip. These sites have a distinct advantage over the email distribution method. First and foremost the shopper is motivated to come to the site, looking for deals. The more relevant the offers and the retailer’s are to the shopper, the better the traffic and the results will be for that site. ... "
You as an Idea Machine
The Ultimate Guide for Becoming an Idea Machine, by James Altucher. A long piece that looks to be worthwhile, just reading. Ideas are the fodder of innovation. " ... You have to come up with a new way of thinking. A new way of having ideas. A new ways of interacting with the outside universe.
You’re in crisis. Time to change. Time to become an IDEA MACHINE. ... "
You’re in crisis. Time to change. Time to become an IDEA MACHINE. ... "
Educating the Big Data Generation
In Fortune: Educating the 'big data' generation ... Classes—and even degree programs—focused on data analytics are cropping up all over the U.S. Behind them? Tech's largest companies.. " With some details about some companies that are selling, buying, and university programs.
I commented in another venue: " ... Yes, the academic connection continues to grow. Though I still see confusion in what it includes. Still need to see more on non big quant capabilities. A big opportunity looms: "Watson, come here, I need you ...? '
I commented in another venue: " ... Yes, the academic connection continues to grow. Though I still see confusion in what it includes. Still need to see more on non big quant capabilities. A big opportunity looms: "Watson, come here, I need you ...? '
Supermarkets and Social Media
In SupermarketNews:
" ... Supermarkets should expand their online presence and include more recipes and meal ideas to connect with consumers beyond coupons and promotional offerings, writes Michael Sansolo, research director for the Coca-Cola Retailing Research Council of North America. Retailers can use social media channels including Pinterest, Snapchat and Instagram to fulfill customer demand for greater brand interaction through cooking tips, inspiration for special events, and exploration of dietary and food trends, he writes. ... "
" ... Supermarkets should expand their online presence and include more recipes and meal ideas to connect with consumers beyond coupons and promotional offerings, writes Michael Sansolo, research director for the Coca-Cola Retailing Research Council of North America. Retailers can use social media channels including Pinterest, Snapchat and Instagram to fulfill customer demand for greater brand interaction through cooking tips, inspiration for special events, and exploration of dietary and food trends, he writes. ... "
Comparing and Contrasting Analytics Types
In Smart Data Collective: I have heard this debated a number of times. Descriptive, Predictive and Prescriptive. Some even add additional types, but I think this is sufficient. And adds a thought: " ... prescriptive analytics compounds the value of predictive through its ability to predict multiple possible outcomes based on the data. By offering a variety of options, prescriptive allows for decision makers to combine the data with their own personal insights, which can lead to better ultimate results. It's reasonable to say that this fact may make prescriptive analytics incredibly attractive to organizations in 2014. ... "
Taxing Enjoyable Work, and Uncanny Valley of Work Related Social Events
Taxing new things, like work satisfaction. I am sure this is all being thought of today. And how do we determine if something is fun, or just engineered to be? And can we can we get refunds for things that are just not enjoyable to us?
Tuesday, May 27, 2014
History and Re-Emergence of Artifical Intelligence
Artificial Intelligence appears to be starting to fuel a new industrial error. Here a nice history of the not too long ago emergence and recession of AI. Course I can give a much more detailed history of its industrial uses, Its been nice to see, under the covers for now, of pieces of the new emergence. And it will be very human, working besides people in creating a new and efficient face to industry. Of course it, as all technology, will have to be carefully managed. " ... Today, artificial intelligence is coming out of the shadows and just about every major tech company has an active artificial intelligence program. From Apple’s Siri to IBM’s Watson to literally hundreds of other applications under development, thinking machines are beginning to permeate our everyday lives. ... "
The NYT on Innovation
Mashable expands on the recent NYT Innovation article. Includes Mashable's takeaways. And a good summary of the challenges related. My only further thought would be that the Times is talking about the challenges today, not what might appear in the next decade. And if that decade is anything like the last, expect the world to change again with challenges anew. Worth reading their current view. With some interesting stats, especially on the falling readership of their home page.
" ... The New York Times commissioned a task force to assess the publication's digital efforts to date and recommend ways it can better keep up with the shifting digital landscape. A portion of the report was was released earlier this month, but on Thursday BuzzFeed obtained a copy of a 96-page version of the report report. ... "
" ... The New York Times commissioned a task force to assess the publication's digital efforts to date and recommend ways it can better keep up with the shifting digital landscape. A portion of the report was was released earlier this month, but on Thursday BuzzFeed obtained a copy of a 96-page version of the report report. ... "
A Laboratory for Innovation
In Innovation Excellence: Another interesting example of a corporate innovation lab. I have reported here many times on the value and direction of the idea. I like the emphasis on disruption, rather than as a means of just testing old ideas. Or just impressing customers. A lab should discover, generate and test ideas. " .... the iLab, Air Liquide ‘lethal weapon’ for breakthrough innovation. ‘Create value by accelerating the pace of innovation’ is its mantra. He kindly took me into a tour to discover his bright Lab, and its sharp activities. .... ". See the link below for more.
Big Data Blog
Tina Groves of IBM has an interesting blog on Big Data, " ... leverages her 25 years in analysis, event processing and information-driven applications to lead the big data product strategy for IBM Business Intelligence. She particularly enjoys sharing how these new technologies can improve decision making with customers. A long-time product manager. she has received corporate recognition for new product introduction and earned two patents. .... "
On Comics and Story Visualization
Via Kevin Winterfield, A look at Scott McCloud's classic book on the comics. The sequential panel model of comics can be used as a means for storytelling a storyboard outline of an idea, often used as a communications approach in advertising. The book is a great look at the history and mechanics of the the method, that we used for early attempts for looking at visual assets.
Deep Brain Implants
Implant charging is a key need for deep brain implants. In Engadget. " ... Stanford researchers have figured out a way to wirelessly charge electronic devices that are deep inside your body. Currently pacemakers and nerve transmitters need to have large receiving coils near the top of your skin to charge up (limiting where you can put them), or periodically have their battery replaced through surgery. This new method, however, would allow devices to be much smaller because they don't have to tote around such a huge battery. Also since they're not reliant on charging coils they could live much deeper in the body and brain. ... "
Monday, May 26, 2014
Strategies for a Digital Age
Recently connected to author David L Rogers. His new book: The Network is Your Customer: 5 Strategies to Survive in a Digital Age. See his site for more information and preview. Planning to read and follow with more later.
Wal-Mart and T-Mobile to Offer Tablets
In CNet: The idea has been pushed since at least 08. It is already being done by retailers in Europe. " ... Walmart and T-Mobile to offer tablets with free data plans .... The self-proclaimed 'un-carrier' is expanding its tablet presence with a rollout of its "free data for life" offer at Walmart stores. ...."
Google Let's you Program a Quantum Computer
Google simulates the programming of a quantum computer. Not create one, but simulate its programming. In ReadwriteWeb. Also, the D-Wave computer, developed by former colleagues, often covered here, is discussed.
" ... Quantum computing has the potential to make of the smartphones, laptops and data centers we now consider so sophisticated look positively Cro-Magnon. The potential is enormous. Have you ever seen the diagram of how many Earths could fit into a planet the size of Jupiter? For comparison’s sake, that is a good place to start.
But for the vast majority of people—including most computer engineers—what quantum computing actually does is basically a mystery. A few “real” quantum computers exist in the world (though some are controversial), and the art of actually programming them to perform useful work is still in its infancy. ... "
" ... Quantum computing has the potential to make of the smartphones, laptops and data centers we now consider so sophisticated look positively Cro-Magnon. The potential is enormous. Have you ever seen the diagram of how many Earths could fit into a planet the size of Jupiter? For comparison’s sake, that is a good place to start.
But for the vast majority of people—including most computer engineers—what quantum computing actually does is basically a mystery. A few “real” quantum computers exist in the world (though some are controversial), and the art of actually programming them to perform useful work is still in its infancy. ... "
Winning Arguments
In Mind Hacks: On the Psychology of winning arguments. " .... How do you change someone’s mind if you think you are right and they are wrong? Psychology reveals the last thing to do is the tactic we usually resort to. ... "
Examples of Biological Swarms
A strikingly photographed piece on examples of animal swarms and their advantages. Recently broadcast on US PBS as 'The Gathering Swarm' and available here. This does not address how we could mimic them, but gives examples of the use of biomimicry. It has been shown that some of these group behaviors can be produced by very simple embedded rules. How can we use that to create collaborative behavior?
Sunday, May 25, 2014
Oculus Rift
In Gizmodo, continued raving for Oculus Rift. Here a hands on look. Still I don't think that most people would be comfortable with the 'strap a screen to your head', extreme 'no hands', 'artificial context' environments. Our own experiment with much earlier ideas led right to vertigo. But I these an eventual future? I think they are." ... So the Oculus Rift is fantastic. If you've used it in its original incarnation, you know that it's incredible. It's virtual reality done better than you've ever seen it before. It's revolutionary. And it's nothing compared to what's coming next. I mean Oh. My. God. Dubbed Crystal Cove, the newest model of Oculus isn't meant for consumers. It's also not just the 1080p upgrade that's been floating around (though it does have a glorious 1080p panel). It's a lot more than that. ... "
Brand Focused CPG Ads
In EMarketer, continued brand focusing of ads:
Study: $4.2B CPG digital ad market focuses on branding
Consumer packaged goods and consumer products companies in the U.S. will spend roughly $4.2 billion on digital media this year, with 65% of that focused on branding and the remainder on direct response, according to an eMarketer report. The brands will spend $1.41 billion on mobile this year, about a third of the overall digital spend .... "
Study: $4.2B CPG digital ad market focuses on branding
Consumer packaged goods and consumer products companies in the U.S. will spend roughly $4.2 billion on digital media this year, with 65% of that focused on branding and the remainder on direct response, according to an eMarketer report. The brands will spend $1.41 billion on mobile this year, about a third of the overall digital spend .... "
Saturday, May 24, 2014
Thinking Swarms
A good overview, non technical article on Swarms. We did some thinking about using robot swarms for repetitive tasks. You can build cheap robot swarms, making them think like swarms that have have useful goals is harder. Preparatory swarms on Mars could be harder yet. A good directional project is the Kilobot project at Harvard. At the right, a Kilobot swarm. Previous pieces in this blog on cooperative swarm technology.
Mining Dark Data
On the potential of dark data. The enterprise often does not know it exists.
" Tapping the profit potential of dark data
Structured and semi-structured data lie dormant in your data warehouse—but it’s becoming increasingly easy to monetize them.
Millions of dollars may lurk somewhere inside your organization’s data warehouses and mainframes. You’ve collected, processed, and stored it; you just haven’t monetized it. You might call it "dark data," and if you can transmute it, the return on investment could be enormous. ... "
" Tapping the profit potential of dark data
Structured and semi-structured data lie dormant in your data warehouse—but it’s becoming increasingly easy to monetize them.
Millions of dollars may lurk somewhere inside your organization’s data warehouses and mainframes. You’ve collected, processed, and stored it; you just haven’t monetized it. You might call it "dark data," and if you can transmute it, the return on investment could be enormous. ... "
Detecting Spurious Correlations
Detailed piece by Vincent Granville. Useful for reference and application.
Tutorial: How to detect spurious correlations, and how to find the real ones
Specifically designed in the context of big data in our research lab, the new and simple strong correlation synthetic metric proposed in this article should be used, whenever you want to check if there is a real association between two variables, especially in large-scale automated data science or machine learning projects. Use this new metric now ... "
Tutorial: How to detect spurious correlations, and how to find the real ones
Specifically designed in the context of big data in our research lab, the new and simple strong correlation synthetic metric proposed in this article should be used, whenever you want to check if there is a real association between two variables, especially in large-scale automated data science or machine learning projects. Use this new metric now ... "
Friday, May 23, 2014
Eric Siegel: Predictive Analytics
Wearable Device Supports Text Reading
(Abstract, full article requires purchase)FingerReader: a wearable device to support text reading on the go
Visually impaired people report numerous difficulties with accessing printed text using existing technology, including problems with alignment, focus, accuracy, mobility and efficiency. We present a finger worn device that assists the visually impaired with effectively and efficiently reading paper-printed text. We introduce a novel, local-sequential manner for scanning text which enables reading single lines, blocks of text or skimming the text for important sections while providing real-time auditory and tactile feedback. The design is motivated by preliminary studies with visually impaired people, and it is small-scale and mobile, which enables a more manageable operation with little setup. ... "
On Practical Machine Learning Problems
Short piece that addresses typical techniques. But it is not all about technique, it is how the technique is connected to the data from the domain of the problem, and ultimately connected to a real world decision.
" ... In this post we will first look at some well known and understood examples of machine learning problems in the real world. We will then look at a taxonomy (naming system) for standard machine learning problems and learn how to identify a problem as one of these standard cases. This is valuable, because knowing the type of problem we are facing allows us to think about the data we need and the types of algorithms to try. ... "
" ... In this post we will first look at some well known and understood examples of machine learning problems in the real world. We will then look at a taxonomy (naming system) for standard machine learning problems and learn how to identify a problem as one of these standard cases. This is valuable, because knowing the type of problem we are facing allows us to think about the data we need and the types of algorithms to try. ... "
Teradata's Data Lab
It is very useful to have a lab to experiment with your data. Here is Teradata's approach.
" ... Promote Exploration & Experimentation
A Teradata Data Lab lets you explore and examine new ideas by combining new data with existing data so its easy to identify new trends and insight or react to immediate business issues. ... "
" ... Promote Exploration & Experimentation
A Teradata Data Lab lets you explore and examine new ideas by combining new data with existing data so its easy to identify new trends and insight or react to immediate business issues. ... "
Thursday, May 22, 2014
Intelligence Matters from O'Reilly
In OReilly: A new blog that appears to be of interest: " ... Today we’re kicking off Intelligence Matters (IM), a new series exploring current issues in artificial intelligence, including the connection between artificial intelligence, human intelligence and the brain. IM offers a thoughtful take on recent developments, including a critical, and sometimes skeptical, view when necessary. ... "
Will start to follow.
Will start to follow.
Proposed Thermal Imaging Wearable AR Headset
New from Metaio. A video describe the future possibilities. Another level of sensor data will be possible. In the press release: " ... With “Thermal Touch”, a wearable headset user could turn any surface into a touch-screen: Imagine pushing directions to your device simply by touching a static map in a shopping mall, building complex or airport; children could bring play to new levels and launch digital content directly from their toys; design professionals could visualize their digital and 3-D creations on their real world counterparts; and service technicians could pull up information just by touching an object in real life.“Thermal Touch” is a prototype and far from everyday usability. Metaio released the demo to educate the community on the possibilities of computer vision. It is likely that in 5-10 years infrared cameras may join a multitude of advanced sensors being integrated into devices everyday, including the wearable augmented reality headsets of the near future. ... "
Real Time Business Intelligence
An important consideration is often what the size of the time step needs to be. I often have to ask, What does your 'real-time' look like? In TDWI. " ... "Real time" of course, means different things depending on users and their circumstances, so expectations need to be managed carefully. Faster always sounds better, and users, being human, are susceptible to the hype of instant gratification. Although Hadoop-related technologies could offer a less-expensive alternative, costs usually rise as organizations push their technology stacks in the direction of real-time data access, analytics, and presentation. Users also need to be informed that data quality will likely fall below standards because there's usually no time to run data quality, profiling, and cleansing processes. ... "
On Domain Names
In Knowledge@Wharton: Interesting how this will change the results. Interesting overview is provided here. Has been a long time since I have looked at domain name generated issues. Consider the ultimately ontology of some of these choices.
" ... Domain registry companies such as Donuts Inc. of Bellevue, Wash., are snapping up names, all for $185,000 a pop, and selling them to registrars who, in turn, are selling web addresses to businesses and organizations that will end with these generic top-level domains.
“The over-arching theme is choice – customer choice, business choice, competition, creativity and choice in the market place,” says James Cole, an ICANN spokesperson. The idea has struck a chord. “We expected to get a few hundred applications,” he notes. In fact, in this first round, it received 1,930 submissions – 911 from North America, 675 from Europe, 303 from the Asia Pacific region, 24 from South America and 17 from Africa. Some of these top-level names are still in development, and others are in dispute. But 263 are already operational, the first buds of what many say is the most significant growth of web properties since the earliest days of the Internet. ... "
" ... Domain registry companies such as Donuts Inc. of Bellevue, Wash., are snapping up names, all for $185,000 a pop, and selling them to registrars who, in turn, are selling web addresses to businesses and organizations that will end with these generic top-level domains.
“The over-arching theme is choice – customer choice, business choice, competition, creativity and choice in the market place,” says James Cole, an ICANN spokesperson. The idea has struck a chord. “We expected to get a few hundred applications,” he notes. In fact, in this first round, it received 1,930 submissions – 911 from North America, 675 from Europe, 303 from the Asia Pacific region, 24 from South America and 17 from Africa. Some of these top-level names are still in development, and others are in dispute. But 263 are already operational, the first buds of what many say is the most significant growth of web properties since the earliest days of the Internet. ... "
IBM Acquires Cognea, Is AI New as Assistance?
This evening listened in on a #Cognitivecomputing presentation given in NYC. See it recorded here. A good broad overview of work done by ETH Zurich and IBM to deliver Watson as a service. See more about their purchase of assistance technology in Cognea. Emphasis is on the virtual assistant model, they say differentiating it from the AI methods of the 90s. I disagree, in the 90s we were not, in industry, attempting to build systems with general human intelligence, but rather methods that were also assistants for narrowly focused domains. Still the piece is worthwhile. I have more notes and may comment on them further. The book at the right was mentioned and covers many of the topics discussed.
Looking at Agile
In CACM: A critical look at Agile. I have taken a look at Scrum for development, but not used it in a real application. And a warning about hype. Bertrand Meyer in a new book. " .... an attempt to sort out the jewels from the gravel. There is so much to be learned from the best agile techniques that it would be a pity to throw them away altogether because of their catastrophic elements. I do not expect everyone to agree with every single one of my assessments, but I hope this attempt at presentation and reasoned analysis will help the software industry benefit from what agile methods genuinely can bring to us. ... " Links to the book.
Wednesday, May 21, 2014
Building a QuBit for a Quantum Computer
In Quanta magazine. A relatively non technical article about building a quantum bit, the Qubit, the foundation for the quantum computer. Nicely described. " ... Willet is attempting to harness that state to build a “topological qubit” — an information-storing device analogous to the bits that make up ordinary computers, only far more complex and potent. Qubits are the basic building blocks of a quantum computer, an undeveloped technology devised in the early 1980s. Unlike that of ordinary bits, the power of qubits grows exponentially with their number. For many tasks, a comparatively small quantum computer — made up of only 100 qubits — would outperform the world’s best supercomputers and usher in a new level of computing power for humanity. ... "
P&G Supply Chain Transformation
Some fairly natural changes. Decrease your transportation lanes. From CGT: " ... P&G Reinforces Supply Chain Transformation ... This innovative approach is expected to increase efficiencies in P&G's supply chain, putting product closer to retail and population centers as well as major transportation networks. ... "
Only One Silicon Valley?
In the NYT: Andreessen on the uniqueness of 'silicon valley', and the future of technologies. I spent much time there, and there was a uniqueness about it I did not see elsewhere. Innovation occurred at all times and in all contexts. It was expected that things that would not work would fall away in a Darwinian way.
" ... The demand for pioneering new technology and an influx of young talent will give rise to new centers of innovation in cities across the world, but according to longtime venture capitalist Marc Andreessen, if they try to be carbon copies of Silicon Valley these new tech hubs will likely fail to gain any traction. Instead, Andreessen says emerging innovation centers should learn from the Bay Area's success and find their own unique niche to define them by focusing on tech-related fields like biotech, digital currencies or virtual reality ... "
" ... The demand for pioneering new technology and an influx of young talent will give rise to new centers of innovation in cities across the world, but according to longtime venture capitalist Marc Andreessen, if they try to be carbon copies of Silicon Valley these new tech hubs will likely fail to gain any traction. Instead, Andreessen says emerging innovation centers should learn from the Bay Area's success and find their own unique niche to define them by focusing on tech-related fields like biotech, digital currencies or virtual reality ... "
Zigbee Alliance
It has been a while, but early in our efforts in promoting an internet of things we looked at the Zigbee Alliance. In particular for future home systems. Have had cause to go back there and examine it again. Worth following.
Data Driven Decisions for Managers
In the HBR: Data Driven Decisions for managers that don't like math. Not uncommon. Useful overview in reading list form. " .... Not everyone needs to become a quant. But it is worth brushing up on the basics of quantitative analysis, so as to understand and improve the use of data in your business. We’ve created a reading list of the best HBR articles on the subject to get you started. ... "
Tuesday, May 20, 2014
CPG Expands Mobile and Location Branding
With some interesting numbers from SmartBlog ...
CPG focuses digital spending on branding, mobile, geotracking
As digital proliferates every facet of the retail industry and consumers’ lives, digital ad spending has grown into a multi-billion dollar business with CPG companies spending more than ever to get their brands and products in front of the right people at the right time. Looking forward into 2014 and beyond, companies will focus on brand messaging and mobile, with an eye on geotracking, according to a new report, The U.S. CPG and Consumer Products Industry 2014 Digital Ad Spending Forecast and Trends, from eMarketer. ... "
CPG focuses digital spending on branding, mobile, geotracking
As digital proliferates every facet of the retail industry and consumers’ lives, digital ad spending has grown into a multi-billion dollar business with CPG companies spending more than ever to get their brands and products in front of the right people at the right time. Looking forward into 2014 and beyond, companies will focus on brand messaging and mobile, with an eye on geotracking, according to a new report, The U.S. CPG and Consumer Products Industry 2014 Digital Ad Spending Forecast and Trends, from eMarketer. ... "
Value and Caution for Data Storytelling
I like the idea, having been introduced to the power of storytelling some time ago. What I do worry about is when the story overwhelms reality. Let's still make it possible to interact with the story and data so it can convince with real results, not fables. The story told should be part of a stream of metadata, not the data itself.
Brain Computer at Sandia Labs
A favorite topic, with a number of labs attempting to address brain inspired neural models. Here one I had not seen as yet at Sandia Labs. " ... Researchers working on a computer that mimics the human brain ... Future computer systems could better tackle big data and autonomous systems .. Researchers at Sandia National Laboratories are working on a computer that can tackle real-world situations in real-time and can run on the same power as a 20-watt light bulb. ... " .
MOOCs not Meeting Expectations
In CACM:
" ... The rollout of MOOCs, or massive open online courses, three years ago by some of the leading universities in the United States triggered predictions that bricks-and-mortar campuses would soon be obsolete and that learning would be forever changed.
Well, maybe — but not any time soon, according to one of the first comprehensive studies of MOOCs from the perspective of institutions, just released by researchers at Teachers College, Columbia University. The study is based on 83 interviews with faculty members, administrators, researchers, and other actors in the MOOC space from 62 institutions, mostly in the United States. It includes 13 case studies to illustrate how MOOCs are successfully being used to address institutional goals. ... "
" ... The rollout of MOOCs, or massive open online courses, three years ago by some of the leading universities in the United States triggered predictions that bricks-and-mortar campuses would soon be obsolete and that learning would be forever changed.
Well, maybe — but not any time soon, according to one of the first comprehensive studies of MOOCs from the perspective of institutions, just released by researchers at Teachers College, Columbia University. The study is based on 83 interviews with faculty members, administrators, researchers, and other actors in the MOOC space from 62 institutions, mostly in the United States. It includes 13 case studies to illustrate how MOOCs are successfully being used to address institutional goals. ... "
Monday, May 19, 2014
New Developments in Eye Tracking Glasses
In Engadget. New advances. in particular a way to get a denser data stream for certain kinds of marketing insight." ... Researchers and marketers often want to know what catches your eye, whether it's players on the basketball court or just a new shampoo at the store. They may have an easier time figuring this out with Tobii's latest eye-tracking headset, the Tobii Glasses 2. Besides providing a much wider field of view than the original eyewear, the new design has a front-facing 1080p camera that lets observers see whatever you're focused on in real time ... "
Future of Computers with Conversational Skills
On Building a Smarter Planet: Future of Computers with Conversational Skills By Mike Rhodin
" ... An essential part of the third era of computing—cognitive computing—will be our ability to interact with smart machines in ways that are more natural to us. Making them conversational is an important part of that effort. ... I’m not talking about just giving the computer a simple command or asking a simple question. That’s yesterday’s technology. I’m talking about more realistic conversations—everything from friendly chitchat to intense debate. As cognitive conversation capabilities advance, we become more than impersonal button pushers. ... In the cognitive era of computing, machines will better understand us and relate to us in more human ways. ... "
The article goes on to state how Watson plans to provide a library of conversational services and supporting knowledge. Ambitious indeed. Having both the knowledge and the ability to establish a conversation about the knowledge, taking the possibilities beyond our current simplistic, imprecise, search-based method for interaction with that knowledge. We have learned how to store knowledge, cognitive aims to bring it to useful life. Conversation lets us tell them what we want.
" ... An essential part of the third era of computing—cognitive computing—will be our ability to interact with smart machines in ways that are more natural to us. Making them conversational is an important part of that effort. ... I’m not talking about just giving the computer a simple command or asking a simple question. That’s yesterday’s technology. I’m talking about more realistic conversations—everything from friendly chitchat to intense debate. As cognitive conversation capabilities advance, we become more than impersonal button pushers. ... In the cognitive era of computing, machines will better understand us and relate to us in more human ways. ... "
The article goes on to state how Watson plans to provide a library of conversational services and supporting knowledge. Ambitious indeed. Having both the knowledge and the ability to establish a conversation about the knowledge, taking the possibilities beyond our current simplistic, imprecise, search-based method for interaction with that knowledge. We have learned how to store knowledge, cognitive aims to bring it to useful life. Conversation lets us tell them what we want.
Programmatic Ad Buying Needs to be Revisited
Prompted by an AdAge article. 'Have begun .. ', the article says? We were early pioneers of programmatic advertising. Starting in the 1970s, using optimization analytics.
" ... The term covers a wide range of technologies that have begun automating the buying, placement and optimization of advertising, replacing human-based methods like phone calls, faxes and, yes, three-martini lunches. Through programmatic technologies, advertisers can buy ads the way they pick up something on Amazon or bid on eBay. ... "
The topic seems rarely covered in analytical writing these days, but is deserving a closer look. At the time we did this there were many millions saved by optimizing placement of ads. Well before big data and newer analytical methods. Ultimately most of this work went out to services, that went back to mostly human powered approaches. This deserves an analytical revisit. Lots of money is still involved.
" ... The term covers a wide range of technologies that have begun automating the buying, placement and optimization of advertising, replacing human-based methods like phone calls, faxes and, yes, three-martini lunches. Through programmatic technologies, advertisers can buy ads the way they pick up something on Amazon or bid on eBay. ... "
The topic seems rarely covered in analytical writing these days, but is deserving a closer look. At the time we did this there were many millions saved by optimizing placement of ads. Well before big data and newer analytical methods. Ultimately most of this work went out to services, that went back to mostly human powered approaches. This deserves an analytical revisit. Lots of money is still involved.
Three Watson Startups Announced
How well a new advanced technology service is working can be measured by how startups are effectively using its capabilities. See the full report. " ... More early-stage companies are signing up for IBM’s fancy Watson supercomputing technology for processing lots of text and then making decisions on the fly. But they’re not rolling in big racks full of servers and hard disk drives to do this. They’re tapping into Watson-as-a-Service. ... On Friday, IBM named three new partners in its burgeoning Watson initiative, following its decision to make Watson available as a managed service. Modernizing Medicine, a company with an electronic medical record system, is using Watson to beef up a new application doctors can use to pull up information while they’re working with patients. ... "
Global Flow Reshaping the World
From McKinsey: Global Flow reshaping the world. Podcast and graphics. We examined some of these concepts in several redesigns of global supply chains. " ... Global flows have been a common thread in economic growth for centuries, since the days of the Silk Road, through the mercantilist and colonial periods and the Industrial Revolution. But today, the movement of goods, services, finance, and people has reached previously unimagined levels. Global flows are creating new degrees of connectedness among economies—and playing an ever-larger role in determining the fate of nations, companies, and individuals; to be unconnected is to fall behind. ... "
Sunday, May 18, 2014
6th International Conference on Concept Mapping
Colleague Brian Moon reports (The link below provides much more)
Deadline Extension - CMC2014
Chief Technology Officer at Perigean Technologies LLC
The deadline to submit papers and posters to the 6th International Conference on Concept Mapping has been extended to Monday May 26.
http://cmc.ihmc.us
Deadline Extension - CMC2014
Chief Technology Officer at Perigean Technologies LLC
The deadline to submit papers and posters to the 6th International Conference on Concept Mapping has been extended to Monday May 26.
http://cmc.ihmc.us
On Inventory Analysis
I spent many years doing inventory analysis type work for big enterprises. This article makes some interesting and obvious arguments. But the basics are correct, it is a key ability of any business that provides physical goods. And it has been done successfully for many years, since WWII at least. The cloud is useful to implement it, as it is with any multi step, multi function IT function.
Quantifying Ourselves
Humans becoming Cyborgs. Simplistic piece in SciAM. We are continuing to quantifying ourselves, but does that mean we are knowing ourselves better? Not without better analytical methods.
CIOs and Open Source
CIOs and Open Source: Just becoming involved with examples where open source choices are in play.
In CWorld: Field Notes: Clueless CIOs aside, open source is 'frightening and fantastic'
Savvy enterprises tap open source as a legacy-refresh strategy
Some companies are using open source to bring their legacy apps up to code, but all too many CIOs are still clueless about how often open source is being used in their own organizations. ... "
In CWorld: Field Notes: Clueless CIOs aside, open source is 'frightening and fantastic'
Savvy enterprises tap open source as a legacy-refresh strategy
Some companies are using open source to bring their legacy apps up to code, but all too many CIOs are still clueless about how often open source is being used in their own organizations. ... "
Connectedness and the Smartphone
In GigaOM: Other ways to connect are appearing. But the functionality of the smartphone remains distinctly powerful and convenient. Until we redefine their operation in new ways.
Saturday, May 17, 2014
Weekly Data Science Resources
Data Science Central puts together a nice list of related papers. Descriptions and links here. Well worth following, including technical and non technical contributions.
Consumer Human Interactions
A report from the recent CHI conference. On playful interactions with things like bubbles. I think game dynamics should have more playful interactions, when most of what we see is competitive interactions. Competition does drive, but so does just plain whimsical play.
Sankey Diagrams
I was reminded again yesterday of the Sankey Diagram. Their use has been influenced by examples in Edward Tufte's graphic design books. The link points to a blog devoted to their use for representing complex data interaction of flows and representing progressive partitions of data. Are they just expansions of the recently much reviled Pie chart? Stephen Few seems to support Sankey use in his classic review of charting complexity. See also Karakotsios's work on using them in Apps. I would like to see easier ways of including them in visualization.
History of Photo-retouching
The extent to which we can trust a given image has changed much. This piece examines the history of the process. Suggesting that most photos today are fiction. I worry too that the same can be said about many data representations as well.
Friday, May 16, 2014
Computational and Cognitive Creativity
An interesting article about the use of systems like Watson to add creative dimensions to what were in the past purely computational problems. We made this observation a decade ago. If you can supply direct computational solutions to simple parts of a problem, you can test many more solutions quickly, adding the ability to create and test quickly. It's another argument for abductive design. Now powered by AI style approaches. Good examples in article.
More on abductive thinking and design.
Update: It was pointed out to me by Brian Moon that the paper linked to just above uses concept maps to illustrate this and related concepts such as 'sensemaking'. Taking a look at papers this article points to.
More on abductive thinking and design.
Update: It was pointed out to me by Brian Moon that the paper linked to just above uses concept maps to illustrate this and related concepts such as 'sensemaking'. Taking a look at papers this article points to.
Expect Labs Mindmeld
Taking another look at Expectlabs. And their Mindmeld work. " ... We are undergoing a profound transformation in how we interact with our devices. Today, users expect that the services they tap into are smarter than ever. Predictive technology satisfies this need by adding a layer of personalization — making each experience with our devices more meaningful. A new report from Business Insider: Intelligence shines a light on this development by analyzing a few of the services that are redefining personalization. Expect Labs is listed as a service that can help developers and marketers leverage this game-changing technology ... "
On Data Science and Big Data in Pharma
New colleague Diego Kuonen, PhD in Statistics - CStat PStat CSci - CEO, CAO at Statoo Consulting, gives a presentation: " .... May 13, 2014, at the `SMi Big Data in Pharma' conference in London, United Kingdom: http://www.slideshare.net/kuonen/big-datadatascience-may2014 - also available at http://www.statoo.com/BigDataDataScience/. Would really appreciate your valuable feedback! ... "
AppInventor and Teaching Logic
On teaching programming, and using it as a method to teach logic in general. Judy Robertson in the CACM.
Thursday, May 15, 2014
New Blog at Computer Now: Business Intelligence
Looks to be well done.
MajorBI-All Things Business Intelligence:
Is Your Company Ready to Deploy Business Intelligence Intelligently?
" ... Although the majority of data-driven enterprises may have or use BI software or systems, a recent survey we took of IT executives revealed that there is a vast difference between having BI and using BI to the company’s full advantage. There is an even larger chasm between the cost to implement BI and the value realized from those efforts. This is not good news for anyone whose bread-and-butter comes from demonstrating measurable results and ROI. But understanding the reasons why most companies aren’t realizing the full value can help plot the course for the change and make the most of the marvelous tool that is business intelligence. .... "
MajorBI-All Things Business Intelligence:
Is Your Company Ready to Deploy Business Intelligence Intelligently?
" ... Although the majority of data-driven enterprises may have or use BI software or systems, a recent survey we took of IT executives revealed that there is a vast difference between having BI and using BI to the company’s full advantage. There is an even larger chasm between the cost to implement BI and the value realized from those efforts. This is not good news for anyone whose bread-and-butter comes from demonstrating measurable results and ROI. But understanding the reasons why most companies aren’t realizing the full value can help plot the course for the change and make the most of the marvelous tool that is business intelligence. .... "
A Look at an Analytics Solutions Cloud Marketplace
The medium size business always needs help with choices. They need easy to make choices that are as inexpensive as possible, and link directly to their current infrastructure. That's what makes the choices less expensive and effective for them. Analytics makes these same choices essential for making the modern business efficient.
" ... This week IBM announced a number of new offerings on the cloud, one of which being the IBM Cloud Marketplace. It was designed to be an easy access, technical shopping mall for all users (business through IT and development) to access solutions and technologies on demand. ... "
Empowering the modern business. Much more about that here. I have started to look at the cloud market place.. Its a good start, but will need some additional work. In particular a more direct link between specific problems and solutions. That's the effectiveness part. The small to medium size business does not have the resources required to do a great deal of tailoring of solutions. The solution need to be quickly adaptable to their infrastructure, without large changes in what they do today.
Having a store to go to is a good start. It also includes some short explanatory pieces about the technology involved, making this store a good place to browse. I would further suggest a glossary of terms that may also not be familiar to the browser. Finally some broad statements of value by industries, likely via surveys, to show a company what their competitors are doing. Giving incentive for browsing the offerings.
This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet. I’ve been compensated to contribute to this program, but the opinions expressed in this post are my own and don't necessarily represent IBM's positions, strategies or opinions. #MidsizeIBM
" ... This week IBM announced a number of new offerings on the cloud, one of which being the IBM Cloud Marketplace. It was designed to be an easy access, technical shopping mall for all users (business through IT and development) to access solutions and technologies on demand. ... "
Empowering the modern business. Much more about that here. I have started to look at the cloud market place.. Its a good start, but will need some additional work. In particular a more direct link between specific problems and solutions. That's the effectiveness part. The small to medium size business does not have the resources required to do a great deal of tailoring of solutions. The solution need to be quickly adaptable to their infrastructure, without large changes in what they do today.
Having a store to go to is a good start. It also includes some short explanatory pieces about the technology involved, making this store a good place to browse. I would further suggest a glossary of terms that may also not be familiar to the browser. Finally some broad statements of value by industries, likely via surveys, to show a company what their competitors are doing. Giving incentive for browsing the offerings.
This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet. I’ve been compensated to contribute to this program, but the opinions expressed in this post are my own and don't necessarily represent IBM's positions, strategies or opinions. #MidsizeIBM
Using Bayesian Methods to Find the Malaysian Airplane
Via the SAS Blog. Good example of predictive analytics with a not generally well known technique for Big Data practitioners. Set in a real world tactical problem. Good practical details. Have written about this method many times in this blog.
How Bayesian analysis might help find the missing Malaysian airplane ....
At the time this blog entry was written, there still appears to be little to no signs of locating the missing Malaysian flight MH370. The area of search, although already narrowed down from the size of the United States at one point to the size of Poland, is still vast and presents great challenges to all participating nations. Everything we’ve seen in the news so far have been leads that turn out to be nothing but dead ends. .... "
How Bayesian analysis might help find the missing Malaysian airplane ....
At the time this blog entry was written, there still appears to be little to no signs of locating the missing Malaysian flight MH370. The area of search, although already narrowed down from the size of the United States at one point to the size of Poland, is still vast and presents great challenges to all participating nations. Everything we’ve seen in the news so far have been leads that turn out to be nothing but dead ends. .... "
Tesco CEO on E-Commerce
Been looking at how retailers address e-commerce futures. Here a piece by the Tesco CEO on e-commerce. Not too unexpected.
" ... Tesco group chief executive Philip Clarke discusses the opportunities that exist at the intersection of media, ecommerce and big data. As the second largest retailer in the world, Tesco is moving in the direction of becoming a media and technology company and the leading multi-channel retailer. The objective of Tesco is to put the customer back in control and enabling the customer to engage and transact with Tesco via his/her preferred channel. The percentage of digital sales is growing and that brings big challenges but also offers huge opportunities. An interesting talk about customer loyalty, delivering personalized experiences and the insights from the data they have that helps Tesco to understand and respond to customers’ lifestyles. ... "
" ... Tesco group chief executive Philip Clarke discusses the opportunities that exist at the intersection of media, ecommerce and big data. As the second largest retailer in the world, Tesco is moving in the direction of becoming a media and technology company and the leading multi-channel retailer. The objective of Tesco is to put the customer back in control and enabling the customer to engage and transact with Tesco via his/her preferred channel. The percentage of digital sales is growing and that brings big challenges but also offers huge opportunities. An interesting talk about customer loyalty, delivering personalized experiences and the insights from the data they have that helps Tesco to understand and respond to customers’ lifestyles. ... "
Machine Learning in Excel
Slide show on a number of techniques, including add-ons to Excel for machine learning. Not much detail, but informative about software I had not seen as yet.
Wednesday, May 14, 2014
Rise of the All Data Enterprise
It has arrived, but many companies are discovering that putting all this data together is harder than you think. Not all is captured, or structured, or even fully understood. Beyond that the piece takes a look at Big Data critically.
ExperienceOne Customer Engagement Solution
Just brought to my attention:
IBM just released ExperienceOne - an analytics and automation-based customer engagement solution that ties together marketing, sales and service. Site.
Have taken only a cursory look so far but the integration of social analytics and SPSS Modeler methods is interesting. We used SPSS Modeler integrated methods for internal decision methods.
IBM just released ExperienceOne - an analytics and automation-based customer engagement solution that ties together marketing, sales and service. Site.
Have taken only a cursory look so far but the integration of social analytics and SPSS Modeler methods is interesting. We used SPSS Modeler integrated methods for internal decision methods.
Morphological Approaches to Engineering Design
A long time correspondent, Tom Ritchey, sends a note about an analytical approach that is little used in enterprise, but should be much better known. Using the structure, or morphology, of a business decision problem to improve it. This is a mostly logical rather than numerical approach. Below a link to a not overly technical example paper on the subject. Also see my previous writings on General Morphology Analysis.
Short description of method.
SweMorph | Swedish Morphological Society
An interesting example of the early application of General Morphological Analysis to engineering is the work of the British Norris Brothers, who in the 1960s worked with the speed racer Donald Campbell to achieve several world records. In this paper, Asunción Álvarez (Inplanta, Madrid) describes their work and its significance within its historical context as an early example of the “morphological approach” to actual engineering design.
You can download: “The Norris Brothers Ltd. morphological approach to engineering design – an early example of applied morphological analysis”, Acta Morphologica Generalis, Vol.3 No.2 (2014),
Short description of method.
An interesting example of the early application of General Morphological Analysis to engineering is the work of the British Norris Brothers, who in the 1960s worked with the speed racer Donald Campbell to achieve several world records. In this paper, Asunción Álvarez (Inplanta, Madrid) describes their work and its significance within its historical context as an early example of the “morphological approach” to actual engineering design.
You can download: “The Norris Brothers Ltd. morphological approach to engineering design – an early example of applied morphological analysis”, Acta Morphologica Generalis, Vol.3 No.2 (2014),
Tuesday, May 13, 2014
Automatic Design: Learning from a Network
The connection between automated configuration and design. Here Cisco is talking about a technical network thing. I am not so much interested in how networks are designed. But I am interested in how design might be automated and it's influence in what that design might need to do. Suppose you want to automate a user interface. Or a pattern recognition system. Are there lessons to be learned here?
Tesco Plans a Smartphone
Retail giant Tesco plans to introduce an Android smartphone this year. In Retailwire. Most interesting part: " ... Tesco's smartphone, and the Hudl, will include a "T-button" to provide instant access to Tesco's digital services, including online shopping for groceries, banking and Blinkbox, the retailer's on-demand music and movie service. A scan-and-pay option is promised to arrive later this year. ... "
Free Analytics Handbook, Part III
As I mentioned, this is less a handbook, and more a set of informative interviews of practitioners and promoters of the topic. Now Tristan Tao writes:" We are excited to share with you the final edition of The Data Analytics Handbook! It includes interviews with Hal Varian, the Chief Economist at Google, Tom Davenport, ranked by Forbes as one of the top 10 Masters of the New Economy, and many more!
Download it for free at www.analyticshandbook.com!
Also if you want to learn data science skills, checkout www.teamleada.com.
Thanks!
Tristan, Brian, & Elizabeth ... "
Analytics for CPG
From Accenture. (link fixed) An interesting overview report with quite a bit of detail. Pointing to a recent study. No specific companies are mentioned, but I do know some of their connections. " ... Consumer packaged goods (CPG) companies realize that deep analytics capabilities can be a key differentiator in today’s competitive marketplace; accordingly, they are making analytics-related investments in talent, tools, data and systems to deliver on that promise. Yet according to recent Accenture research, many companies struggle to maximize value from these investments—in part, because of the inability to take a coordinated, enterprise-wide approach supported by a clearly defined operating model. ... "
Drone with Oculus Support
IEEE Spectrum reports on a natural progression, the interaction of physical devices, like drones, and Augmented reality interfaces, like the Oculus Rift. display. The connection between our senses and mobile sensory devices. A way of extending your mobile range, with many of the issues about privacy and regulation. We will see more of this developing. With video and many images.Management Best Practices at GE
Have always been impressed in my occasional collaboration. Includes some interesting examples. " .... A fundamental challenge of any firm – especially a huge global company such as GE – is how to balance nurturing tomorrow’s future businesses, with the resource demands for running and improving today’s operations. You need to think like a portfolio manager, allocating resources both to innovate in your core and for the future. Knowing that today’s operations will almost always win the lion’s share of resources, you need to consciously create a protected class of innovative ideas to invest in, even if money is tight. .... " In the HBR.
Monday, May 12, 2014
Understanding Faces and Consumer Behavior
Understanding consumer behavior by building on facial recognition software. Work that came out of MIT where we saw it being used to trap and classify emotional responses to products via a set of digital eye glasses. The first step into what they called affective computing. How will this then be compared to 'deeper' neural based methods? Am currently looking at several comparable methods.
Smart Home Appliances from LG
An area we examined for years. How could smart appliances be integrated into the home? And what are the implications for the large CPG company? A short look at how always innovative Korean appliance manufacturer LG is bringing out new networked appliances, and controlling them with a HomeChat Service, based on a texting communications model. More on HomeChat. Claims are that other home services will also be linked to the model. And of course, what will people pay for this? Video at CNET.
Smart Look at Hadoop and Big Data
Good piece by Tina Groves. I have discovered that many technologists, especially in the small to medium sized businesses, do not understand the value Hadoop or Big Data. Or how the methods work together with predictive analytics. Here is a perspective from IBM, using their suite of tools. Not universal, but useful to review.
It is also key to understand that analytics methods are most purposefully applied against decision processes. They do not always need complex methodologies, but start with the simplest techniques, like visualizing data that is created by your methods. Groves writes:
" .. For many people, big data is synonymous with Hadoop. Certainly, the ability to store and processing vast amounts of data on commodity hardware has fueled a new generation of applications. The synergies of Hadoop, fast and reliable internet and the increasing love for all things mobile have channeled heady investments in software as a service (SaaS) startups.
IBM understands the opportunity with Hadoop and addressing big data challenges. Early and continued investment in this area is why IBM has been consistently rated as a leader in numerous analyst reports, including the recent Forrester Wave on Big Data Hadoop Solutions. ...
In the big data realm "data” means obtaining data as close as possible at the source, and then analyzing it for immediate action. IBM® InfoSphere Streams enables tapping into streaming data or data in motion where the throughput rates are from the thousands to millions per second. Whether that data originates from medical devices to monitor neonatal infants or from sensors to predict manufacturing yields, InfoSphere Streams has opened the door to exciting new applications of analytics. ... "
This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet. I’ve been compensated to contribute to this program, but the opinions expressed in this post are my own and don't necessarily represent IBM's positions, strategies or opinions. #MidsizeIBM
It is also key to understand that analytics methods are most purposefully applied against decision processes. They do not always need complex methodologies, but start with the simplest techniques, like visualizing data that is created by your methods. Groves writes:
" .. For many people, big data is synonymous with Hadoop. Certainly, the ability to store and processing vast amounts of data on commodity hardware has fueled a new generation of applications. The synergies of Hadoop, fast and reliable internet and the increasing love for all things mobile have channeled heady investments in software as a service (SaaS) startups.
IBM understands the opportunity with Hadoop and addressing big data challenges. Early and continued investment in this area is why IBM has been consistently rated as a leader in numerous analyst reports, including the recent Forrester Wave on Big Data Hadoop Solutions. ...
In the big data realm "data” means obtaining data as close as possible at the source, and then analyzing it for immediate action. IBM® InfoSphere Streams enables tapping into streaming data or data in motion where the throughput rates are from the thousands to millions per second. Whether that data originates from medical devices to monitor neonatal infants or from sensors to predict manufacturing yields, InfoSphere Streams has opened the door to exciting new applications of analytics. ... "
This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet. I’ve been compensated to contribute to this program, but the opinions expressed in this post are my own and don't necessarily represent IBM's positions, strategies or opinions. #MidsizeIBM
Big Data and Social Physics Course
Of interest, can be audited free.
" ... Social physics is a big data science that models how networks of people behave and uses these network models to create actionable intelligence. It is a quantitative science that can accurately predict patterns of human behavior and guide how to influence those patterns to (for instance) increase decision making accuracy or productivity within an organization. Included in this course is a survey of methods for increasing communication quality within an organization, approaches to providing greater protection for personal privacy, and general strategies for increasing resistance to cyber attack. ... "
" ... Social physics is a big data science that models how networks of people behave and uses these network models to create actionable intelligence. It is a quantitative science that can accurately predict patterns of human behavior and guide how to influence those patterns to (for instance) increase decision making accuracy or productivity within an organization. Included in this course is a survey of methods for increasing communication quality within an organization, approaches to providing greater protection for personal privacy, and general strategies for increasing resistance to cyber attack. ... "
Shopping Image Recognition by Amazon: Flow
Have spent some time on the process of shopping using in-aisle using scanning methods. Finally got a chance to look at this method more closely using an IOS installed Amazon App: Flow. The approach, a little mysteriously, is called Flow. I imagine implying how it could improve the flow of the in-store shopping experience. Here is Amazon's description. The camera is used as a means of reading symbol codes and images. See my previous notes on this.I am working with a used book selling operation, and its useful to be able to quickly scan and recognize inventory. Some time ago the process moved to scanning book barcodes and getting meta information from the Amazon database.
This test went against a number of books, variously oriented, without searching for a barcode first. The system works quite well, faster than a barcode read. Reading from the spine side of a book was only sometimes successful.
Overall well done, can improve interaction in store and warehouse. Woould be better yet if it could read spine directed books. We were also able to identify several items in a single view, but that is not currently an operational need.
Sunday, May 11, 2014
Guy Kawasaki on the Art of Innovation
Guy Kawasaki on Innovation
' ... “Innovation is the specific instrument of entrepreneurship … the act that endows resources with a new capacity to create wealth.” — Peter Drucker
Peter Drucker was right: Innovation is what creates wealth. Innovation had better create wealth because it’s so damn hard to do. The purpose of this piece is to explain the principles of innovation I learned in the trenches of Apple Inc., the three companies I started and the dozens of companies I’ve advised. I wish I could tell you I did all these things right. That would be a lie. Do as I say — not as I did. ... '
' ... “Innovation is the specific instrument of entrepreneurship … the act that endows resources with a new capacity to create wealth.” — Peter Drucker
Peter Drucker was right: Innovation is what creates wealth. Innovation had better create wealth because it’s so damn hard to do. The purpose of this piece is to explain the principles of innovation I learned in the trenches of Apple Inc., the three companies I started and the dozens of companies I’ve advised. I wish I could tell you I did all these things right. That would be a lie. Do as I say — not as I did. ... '
Press Explaining News with Technology: Vox
How much should the press explain the news rather than just present it? A colleague pointed out Vox to me. Positioned as a closer melding of technology and journalism. Is that a good thing? It is inevitable. What will it mean to advertising and promotion? Much.
Is the Data Analytics Field Consolidating?
First I heard of this. Not sure I agree. In GigaOM
" ... The big data and analytics world is entering that consolidation phase right now. Buyers can’t eliminate risk, but they should watch market consolidation in order to mitigate it and formulate contingencies. ...
Most technology sectors go through a certain maturity cycle. At the beginning of the cycle a number of pure-play companies, flush with venture capital in their pockets, come in for the gold rush. Next, a few of them crash and burn while a few others gain revenue momentum and significant name-recognition. Meanwhile, the majority of companies tend toward the neither extreme and just “keep on truckin.” ... '
Have seen some of the small examples mentioned, but the big companies are still fighting it out. Even in large companies, where they often coexist. Most often they still fight with existing efforts in general purpose tools like Excel.
" ... The big data and analytics world is entering that consolidation phase right now. Buyers can’t eliminate risk, but they should watch market consolidation in order to mitigate it and formulate contingencies. ...
Most technology sectors go through a certain maturity cycle. At the beginning of the cycle a number of pure-play companies, flush with venture capital in their pockets, come in for the gold rush. Next, a few of them crash and burn while a few others gain revenue momentum and significant name-recognition. Meanwhile, the majority of companies tend toward the neither extreme and just “keep on truckin.” ... '
Have seen some of the small examples mentioned, but the big companies are still fighting it out. Even in large companies, where they often coexist. Most often they still fight with existing efforts in general purpose tools like Excel.
Simple Data Visualization for Retail
Recall I mentioned this effort here before. Now a reminder from the HBR:
' ... “Retailers are all using scanner data to track what happened at the point of sale,” says Sam Hui, an associate professor of marketing at NYU’s Stern School of Business. “But they have no idea what’s really happening at a point-of-purchase decision.”
This is changing with the emergence of location analytics. Take Alex and Ani, which designs and retails jewelry, and Belk, a department store chain. Both have signed on with Prism Skylabs, a software company, to map in-store customer behavior.
By using a store’s existing security cameras, or installing new ones, Prism (no relation to the NSA program) is able to track the movement of a store’s customers and identify patterns. “We’re not really looking at any individual; we’re looking at what a group of people over a period of time do,” says senior vice president of managed services Cliff Crosbie. “That’s the really big thing: Identifying what a volume of people do over a period of time, and how you read that information.” ... '
' ... “Retailers are all using scanner data to track what happened at the point of sale,” says Sam Hui, an associate professor of marketing at NYU’s Stern School of Business. “But they have no idea what’s really happening at a point-of-purchase decision.”
This is changing with the emergence of location analytics. Take Alex and Ani, which designs and retails jewelry, and Belk, a department store chain. Both have signed on with Prism Skylabs, a software company, to map in-store customer behavior.
By using a store’s existing security cameras, or installing new ones, Prism (no relation to the NSA program) is able to track the movement of a store’s customers and identify patterns. “We’re not really looking at any individual; we’re looking at what a group of people over a period of time do,” says senior vice president of managed services Cliff Crosbie. “That’s the really big thing: Identifying what a volume of people do over a period of time, and how you read that information.” ... '
Saturday, May 10, 2014
Historical Look at Netscape
Our labs were early users of the Mosaic browser to communicate with University research. We watched the work of Jim Clark out of SGI with Marc Andreessen. We used browsers like theirs for our Intranet. So it is interesting to look at the history of fundamental Internet tools, like the browser. Why did some succeed, and others like Netscape fail?
Synthetic Biology Conference
In the NYT: A topic we linked with the Santa Fe Institute on. Also with MIT prof Neil Gershenfeld, who is quoted in the article. " Neil Gershenfeld, ... a physicist who is the director of the Center for Bits and Atoms, said that the improvement in the capacity to read and write biological genes has given rise to the possibility of “spectacular advances,” like the ability to use a computer to design a complete genome, output it, insert it in a cell and in effect create life from scratch.The new abilities, he noted, raised ethical questions that are as yet unanswered. “When the ability to convert biology to data and data into biology becomes that cheap, that agile, that easy to do, what are the consequences?” he said. ... "
We saw this as more directional for modeling capabilities, based on biological examples, rather than about creating life. Often using agent models inspired by biology and even swarms of animals.
Correlation does not mean Causation
A group of amusing slides that make the case. Probably best as examples for the non scientist managers you work with. But you would be surprised how much effort has been hung on a good correlation number by scientists. Even with all the 'causation is not necessary' talk of late. A good data point of caution resides here.
Basic Tools for Data Viz
From Vincent Granville in Data Science Central: 30 Basic Tools for data visualization. Software and other resources. No reason anyone should not be able to deal with visualization needs. Nice summary of links.
Shopping by Scanning Images
In MarketWatch: We experimented with approaches that used watermarks embedded in packaging. Includes a short video demo to show it in operation.
" .... You can now shop on Amazon by taking photos
App’s new “Flow” feature allows for point-and-shoot shopping
" ... Amazon app's image recognition
Amazon is integrating product recognition software, called “Flow,” into its main shopping app. Point your phone’s camera at a labeled product, and if Amazon sells it, you’ll see it pop up.
Flow—as its name suggests—aims to make it as seamless as possible to shop. MarketWatch carried out its own “showrooming” with the app. Scanning a three-bottle package of the hair growth serum Rogaine, Flow immediately found the item on Amazon for $43.85, 30% cheaper than the $62.99 price in a Duane Reade store. .... "
" .... You can now shop on Amazon by taking photos
App’s new “Flow” feature allows for point-and-shoot shopping
" ... Amazon app's image recognition
Amazon is integrating product recognition software, called “Flow,” into its main shopping app. Point your phone’s camera at a labeled product, and if Amazon sells it, you’ll see it pop up.
Flow—as its name suggests—aims to make it as seamless as possible to shop. MarketWatch carried out its own “showrooming” with the app. Scanning a three-bottle package of the hair growth serum Rogaine, Flow immediately found the item on Amazon for $43.85, 30% cheaper than the $62.99 price in a Duane Reade store. .... "
Friday, May 09, 2014
Visualizing Data from High Performance Computing
Simulation creates a great deal of data, so it naturally links well with visualization Every industrial simulation we wrote for the enterprise was linked to a visualization to understand the subtleties of parameter changes and experimentation. In Computer Org: Largely non technical coverage." ... For computational simulations, the era of "big data" ended before it began. We're actually living in the era of infinite data — in which the stream pouring forth from computational models and simulations can be as voluminous as outputing every value at every timestep, drowning disks and researchers alike with high-cadence, arbitrarily large datasets. Rather than struggling to make models bigger, the challenge is now to keep them under control. ... "
Links to a number of other articles and resources. See link below for more coverage in this blog on HPC.
Big Data and Food and Beverage Retail
A Podcast of interest.
" .... Nearly every company wants to acquire more customers and keep them longer. Yet in some ways, that’s more challenging now than ever with so many options available to consumers. But there is also tremendous opportunity for companies that are using data and analytics in creative and innovative ways. Michael Haydock, IBM Fellow, Distinguished Engineer and Chief Scientist of the Business Analytics and Optimization group, described how one popular food & beverage retailer is marrying billions of transactional records with weather and social media data to create highly personalized promotions. ... "
" .... Nearly every company wants to acquire more customers and keep them longer. Yet in some ways, that’s more challenging now than ever with so many options available to consumers. But there is also tremendous opportunity for companies that are using data and analytics in creative and innovative ways. Michael Haydock, IBM Fellow, Distinguished Engineer and Chief Scientist of the Business Analytics and Optimization group, described how one popular food & beverage retailer is marrying billions of transactional records with weather and social media data to create highly personalized promotions. ... "
Profiling via Shopper Self Scans
In a Telagraph piece, the suggestion that data gathered from self scanning checkouts could be used to profile shoppers. Fairly thin article. Points to a Symbol Technologies Patent. That points to some specific statistical methods used.
Google Acquires Rangespan
Addressing supply chain management. " .... Google has made another move to build out its e-commerce business, and specifically its retail portal Google Shopping. It has acquired Rangespan, a London-based provider of back office services for online retailers, using data science to help them expand their product selection based on real-time sales dynamics. ... "
Big Data Viewed by Philosophers of Mathematics
In a very philosphical, long, but non technical piece from the AMS, provocative. A methodological danger? Worth reading.
Domenico Napoletani, Marco Panza, and Daniele C. Struppa
" ... The methodological danger is that the flood of data generated by our innumerable measuring devices may convince us that data is enough; that there is nothing beyond the microarray paradigm; and that opaque, enormous, data-driven models are the privileged way to approach phenomena, even though they become so similar to the famous map of Borges [4], which was useless, since it was as big as the geography it was supposed to describe. .... "
Thursday, May 08, 2014
Truth Box
Dan Ariely presents the traveling truth box. How might this create a baseline for understanding our conception of truth? How does it change the behavioral economy? " .... The Truth Box is a traveling story booth where we invite participants to share the truth about a lie they’ve told. These personal stories will be recorded and ultimately shared online. The Truth Box installation recently premiered at the Games for Change Festival, and is now in YouTube’s reception area in the Chelsea Market (75 9th Avenue) in New York City. If you are in the area, stop by and share your truth! ... "
Data Visualization: Food Availability
Another interesting Tableau example. Here food availability. I like the interactive aspects of the visual. " ... Graham Douglas of The Economist shows us how food availability measured in kilo-calories has changed since 1961. Use the slider to pick a decade and then hover over countries to see the food calories available per person ... ". The gallery contains many other examples.
Life Forms in the Internet of Things
Security chief of In-Q-Tel suggests that entities in the Internet of Things are life forms. " ... They are immortal in the sense that they can continue to function for years at an assigned task. "The longer lived these devices," said Geer, "the surer it will be that they will be hijacked within their lifetime." ... "Their manufacturers may die before they do -- a kind of unwanted legacy much akin to superfund sites and space junk," said Geer. So something has to be done. ... " .
Watson Cognitive to be Included in University Curricula
Press release: " ... In Fall, 2014, New Courses Will Inspire University Students to Build Apps Infused with Watson's Intelligence While Gaining the Entrepreneurial Vision to Deliver Their Innovations Into the Marketplace ... Announcement Marks the Newest Step in IBM's Strategy to Fuel an Ecosystem of Innovators Who Will Make Cognitive Computing the New Worldwide Standard of Computing ... "
As I have mentioned here before, there is opportunity to integrate cognitive intelligence with analytical intelligence. Where better to do that than via research in a university? Hope to have more details to report here as this progresses.
" ... For the first time, enrollment is now open for fall 2014 cognitive computing courses at Carnegie Mellon University, New York University (NYU), The Ohio State University, Rensselaer Polytechnic Institute (RPI), University of California, Berkeley, University of Michigan and the University of Texas in Austin. ... "
Wednesday, May 07, 2014
Data Provenance for Analytics
A short piece in the CACM that points to a larger paper. That larger paper requires subscription. I am using this for some work on metadata uses. Provenance is tracking some entity, such as data, over time to understand where it has come from and crucially the circumstances under which the data was gathered. Often key to using it for predicting things using analytics. Nicely done paper I am bookmarking for a larger effort.
Full Abstract:
" ... Assessing the quality or validity of a piece of data is not usually done in isolation. You typically examine the context in which the data appears and try to determine its original sources or review the process through which it was created. This is not so straightforward when dealing with digital data, however: the result of a computation might have been derived from numerous sources and by applying complex successive transformations, possibly over long periods of time.
As the quantity of data that contributes to a particular result increases, keeping track of how different sources and transformations are related to each other becomes more difficult. This constrains the ability to answer questions regarding a result's history, such as: What were the underlying assumptions on which the result is based? Under what conditions does it remain valid? What other results were derived from the same data sources? .. ."
Full Abstract:
" ... Assessing the quality or validity of a piece of data is not usually done in isolation. You typically examine the context in which the data appears and try to determine its original sources or review the process through which it was created. This is not so straightforward when dealing with digital data, however: the result of a computation might have been derived from numerous sources and by applying complex successive transformations, possibly over long periods of time.
As the quantity of data that contributes to a particular result increases, keeping track of how different sources and transformations are related to each other becomes more difficult. This constrains the ability to answer questions regarding a result's history, such as: What were the underlying assumptions on which the result is based? Under what conditions does it remain valid? What other results were derived from the same data sources? .. ."
Subscribe to:
Posts (Atom)