Bots and Spam

Very interesting historical and technical view of Spam.   First the history and then the look at how the technical model of a markov chain is involved. Great example to think about how state models work.  And how these models start to look like the basic intelligence of Bots.

The claim is made that the first commercial Spam was sent in 1994.  That made me think, I am sure I saw Spam on groups well before then.  But then it may hang on what 'commercial' means, and the volume involved. Well worth a read from the technical perspective, has given me a few ideas already.

Spam Decreasing

At a twelve year low as we learn to detect it better, but Malware continues to rise and is harder to detect.

Google Catching GMail SPAM

In Wired: Spam does seem like a far less common occurrence these days.   Using neural methods. Now can we address other forms of malware?

" ... Three years after it last released Gmail’s spam stats, Google says that its spam rate is down to 0.1 percent, and its false positive rate has dipped to 0.05 percent. The company credits the significant drop in large part to the introduction of brain-like “neural networks” into its spam filters that can learn to recognize junk mail and phishing messages by analyzing scads off the stuff across an enormous collection of computers. ... "

Fridge Caught Sending Spam

From CNet
Fridge caught sending spam emails in botnet attack ... In the first documented attack of its kind, the Internet of Things has been used as part of an attack that sent out over 750,000 spam emails...... " 

How Enterprises Use AI

In CWorld:   Good overview of the expansion of the use of cognitive, AI and links to Machine learning.  Based on learnings at the recent AAAI meeting.   My experience and connections, have led to getting lots of queries about where to start.

" .... While A.I. is being used in smartphones and self-driving cars, it's also working its way into the enterprise to filter spam out of email, handle complicated scheduling or detecting fraud in big data deployments.

"I think IT probably needs artificial intelligence," said Stephen Smith, a professor who specializes in robotics and A.I. at Carnegie Mellon University. "There are increasing cyberattacks we're dealing with. We have bigger, more complex problems with all the issues arising out of the explosion of the Web and all of our big data. We're already using A.I. It's already there. I think A.I. in the enterprise is going to start to cascade." ... " 

Sample Naive Bayes Text Classification in Python

Via O'Reilly: Nice Python example of using Naive Bayes for text classification.  With actual code, and links to talk and slides.

" ... Full repo here: https://github.com/arnicas/NLP-in-Python
This is an example of going from labeled text to machine classification, first with NLTK and then the Python machine learning library scikit-learn. Examples updated from my OpenVis Conf talk here, which is more entertaining: https://www.youtube.com/watch?v=f41U936WqPM and slides: http://www.slideshare.net/arnicas/the-bones-of-a-bestseller
Warning: Rated NC-17. Using text samples from "50 Shades of Gray"! (Because spam is boring.) ... " 

Comment Requirements Changed

Due to increasing Spam in comments,  you now need to register to post a comment.  In addition, as before, all comments are moderated.  Historically this blog has few comments. I still welcome your thoughts, my address can be found in a number of locations.  I will answer any serious questions.  I also sometimes repeat a conversation here.

Twitter Classifications

As part of a recent project about twitter feed relevancy. I found this short mention of content classification of Tweets in the wikipedia.    See the link for further references:

"Content Classification

San Antonio-based market-research firm Pear Analytics analyzed 2,000 tweets (originating from the US and in English) over a two-week period in August 2009 from 11:00 am to 5:00 pm (CST) and separated them into six categories:[90]

Pointless babble – 40%
Conversational – 38%
Pass-along value – 9%
Self-promotion – 6%
Spam – 4%
News – 4% ... "

Developing Analytic Talent

Just received.  A quick look indicated that this is a useful, only mildly technical look at what is being called Data Science.  A guide and handbook for anyone interested into the topic. Will review further and post that here soon.

Developing Analytic Talent: Becoming a Data Scientist
by Vincent Granville

Their decription:  " ... Harvard Business Review calls it the sexiest tech job of the 21st century. Data scientists are in demand, and this unique book shows you exactly what employers want and the skill set that separates the quality data scientist from other talented IT professionals. Data science involves extracting, creating, and processing data to turn it into business value. This guide discusses the essential skills, such as statistics and visualization techniques, and covers everything from analytical recipes and data science tricks to common job interview questions, sample resumes, and source code.

The applications are endless and varied: automatically detecting spam and plagiarism, optimizing bid prices in keyword advertising, identifying new molecules to fight cancer, assessing the risk of meteorite impact. Complete with case studies, this book is a must, whether you're looking to become a data scientist or to hire one. ... " 

Attributing Attacks on the Internet of Things

A discussion of the implication of security on the Internet of Things.  If everything is connected, the path to us or our households brings insecurity yet closer.     So can we attribute the source of recent attacks?  " ... Symantec published a response indicating that IoT devices were not responsible for the spam attack in question, and the machines behind the spam attack were all really just infected Windows boxes. So why is determining the identify of the devices used in this spam attack so difficult? ... " 

Comments on this Blog

Repeating my policy.  Readership is up and Spam is increasing.  All comments are moderated. I cannot read or approve anonymous comments without prior approval.  I cannot approve any comment that is not substantive to the message it comments on.  I can read only comments in English or German.

Reacting to Google Algorithm Changes

Another Algorithm Change.  Is there no other feasible reaction to the ongoing changes by Google?

" ... Every online publisher and her brother has since rushed to spill ink on the topic because we all want to know: how can we please Google? Not like souless robotic spammers, of course. But like people who sincerely believe they deserve to rank highly (even if that becomes a less meaningful metric by the day). Because, well, think about it …

If we all knew the ins-and-outs of how Google ranks content, and if we knew which factors they use and the exact weight each carries in ranking, then all we’d have to do is just tick off our check list, publish the post, and bask in the light of steady traffic pouring in from our high rankings! .. " 

Topology Mapping for Analytics

Seems Vincent Granville has been thinking similarly lately.  See his article and code in Data Science Central:   A little known component that should be part of most data science algorithms. Beware, this has a general introduction, and then gets technical with R code. And goes further:

" ...  This is a component often missing, yet valuable for most systems, algorithms and architectures that are dealing with online or mobile data, known as digital data: be it transaction scoring, fraud detection, online marketing, marketing mix and advertising optimization, online search, plagiarism and spam detection, etc. .... I will call it an Internet Topology Mapping. It might not be stored as a traditional database (it could be a graph database, a file system, or a set of look-up tables). It must be pre-built (e.g. as look-up tables, with regular updates) to be efficiently used. ... "

I came to think of this when I experimented with the free, open source GePhi network management system and reported on it here. Though the most common network we might work topologically is the Internet, or a social network. These are not the only networks, things like influence maps come to mind.     Have not thought how R in particular links with GePhi, but exploring.  I have a real application in mind. Any thoughts out there?

Semantic Startup for Content Generation

We experimented with methods like this, first I have seen it in the wild again.  A means of combining human and machine skills.

Analytically generating content based on data .....
I asked Musgrove whether this kind of automated content generation is potentially spammy and whether he sees automation replacing the work of humans.

“We seldom really generate something completely artificially,” he replied. “You’re always leaning on human beings in the beginning.” To illustrate that point, he said the suggested headlines are usually phrases pulled from the articles that seem particularly “intense” and have been translated into “headlinese”: “It’s not as artificial as you would think.” .... 

Business Analytics in Retail: The Book

Large companies can afford the ability to use consultants who can help them understand such terms as 'Business Intelligence'. 'Predictive Analytics', "Big Data'  or 'Data Mining'.  But the small to medium size company often needs some basic help with the terminology, which is often changing quickly.  One approach is to read blogs like this, which cover the terms, methods and vendors that play in these spaces.  This also gets you access to many years of experience in the broad area of 'business analytics'.

But where do I get a basic introduction to this space at minimal cost?  I just had the opportunity to read a book created by IBM: Business Analytics in Retail for Dummies  " Read this ebook to learn how analytics can improve every aspect of your business... "     You can get the book free here.  You just have to provide some basic registration information.  You won't get a call or spam unless you ask for it

Despite cringing at the 'Dummies' label, we know you are not one, you are an expert at your business. This free, short 75 page e-book does a very good job of defining the terms and giving some excellent examples of the use of analytics.  Well written and concise.  Although the examples are all about retail, it is worth reading for some of the definitions.   Pointers to IBM software, of course, but there are other places to go as well.   I think this book should be on everyone's e-shelf.  Use the definitions and examples with your employees for hints at for where to apply business analytics.  Save money and increase profit as a result.

This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines  of a smarter planet. More here.  

Commenting on Communications

I am getting increasing number of Spam Comments with an author of  'Anonymous'.  It is getting difficult to even scan all messages.  If you would like to improve your chances of being approved for publication include an author name.  If you want to get to me privately for a speaking or consulting session, use the contact information in the left hand column.  I am speaking at the Rotman School of Business in Toronto this week.  I will always respond to requests promptly.  I will not approve comments that are not on point to this blog and the message commented on.

Big Data, Smaller

In Wired:   Dremel vs Hadoop in big data exploration and leverage.  Good read. " ... Dremel is a way of analyzing information. Running across thousands of servers, it lets you “query” large amounts of data, such as a collection of web documents or a library of digital books or even the data describing millions of spam messages. This is akin to analyzing a traditional database using SQL, the Structured Query Language that has been widely used across the software world for decades. If you have a collection of digital books, for instance, you could run an ad hoc query that gives you a list of all the authors — or a list of all the authors who cover a particular subject. ... " 

Comment Thoughts

Anyone who runs a blog has noticed that there are sharply increasing amounts of Spam being received.  I approve all comments to this blog.  I cannot approve anything that is written in any language but English and German.  So if it is that important, have it translated.  Also I will not approve anything that does not contribute to the conversation substantially.  'Nice post' does not qualify.  I will only rarely approve anything with a link in it, unless I am sure of the source.   If you want to personally communicate with me, sending a note to my comments asking to communicate sometimes works, but also sometimes gets lost in the traffic. Use my email address. There are also some trolls out there that seek to harass me via comments, shame on you.