Ran into exactly this problem recently. Mostly we solved using R as an abstraction layer with sampled data. There are a number of other solutions in the article:
You’ve got three options: Scaling up, scaling out, or using R as an abstraction layer.
By Federico Castanedo
For more on this topic, Brian Kreeger and Roger Fried will be hosting a live webcast, Scalable Data Science with R, on August 16, 2016.
R is among the top five data science tools in use today according O’Reilly research; the latest kdnuggets survey puts it in first, and IEEE Spectrum ranks it as the fifth most popular programming language.
The latest Rexer Data Miner survey revealed that in the past eight years, there has been an three-fold increase in the number of respondents using R, and a seven-fold increase in the number of analysts/scientists who have said that R is their primary tool.
Despite its popularity, the main drawback of vanilla R is its inherently “single threaded” nature and its need to fit all the data being processed in RAM. But nowadays, data sets are typically in the range of GBs and they are growing quickly to TBs. In short, current growth in data volume and variety is demanding more efficient tools by data scientists ... "
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment