Data Science Lessons: Insights from an Agricultural Proof of Concept

Agriculture has come a long way from ancient times through the industrial revolution to the current digital era. In 2017, modern agricultural organizations have access to increasingly large amounts of data collected by sensors from soil quality measurements, weather sensors, GPS guided machinery, and more. According to a USDA’s recent survey, more than 60 percent of corn and soybean crops are monitored by data collection devices. However, there is still a substantial gap between the potential of utilizing this data and what happens in reality. Despite having the data, many companies lack the capability to effectively process, analyze, and efficiently build informative models in order to make data-driven decisions.

That’s where guidance from data service providers, such as Virtustream, can help. Virtustream provides data management expertise, tools and data science consulting to enable customers across different industries to get value from their data resources.

Our data science team in Dell IT recently initiated a Data-Science-as-a-Service Proof of Concept (PoC) as part of a Virtustream service engagement with a large company that plants thousands of farms across USA. Virtustream had enabled the company to become more data-driven by harnessing its large amounts of data, as well as developing and implementing different applications that enable scalable, faster, and more accurate operations – operations that couldn’t be executed with existing tools. Our PoC sought to demonstrate the speed and efficiency of those analytics applications.

Our goal in this PoC was to enable fast and automated execution of seasonal yield predictions for each field in terms of tons per acre using Virtustream cloud services. Such a prediction has a significant business value for the company as it enables efficient resource allocation among the different fields or, in other words, maximal crop yield with minimal costs.

In a three week sprint, we delivered a model that is at least as accurate and is much faster than the existing model (which runs on a single lap-top). The model calculated the predicted yield of a given field based on internal data sources (soil type, fertilizers, plant ages, etc.) and external public sources like satellite images and historical weather data. This short-term engagement highlights four points that we find worth sharing with readers:

Data Science as a part of a bigger solution

Cloud and storage services serve as tools to achieve what the customer really wants – business value. A key way to illustrate that business value to the customer is via Data Science PoC outputs that provide concrete proof of what can be achieved. We request the customer to share a dataset (or a fraction of it) and after several days we deliver concrete evidence of the potential ROI that can be gained with the proposed solution. In the discussed PoC, we proved that using the proposed Virtustream cloud infrastructure, a prediction model that use to take several days to create may be built in less than 30 minutes.

There is a fine line between Small Data and Big Data

‘Big Data’ has become one of the most widespread buzzwords of this decade. But what is Big Data? Is it a 2GB dataset, 100GB or 1TB? From our perspective, the answer is very clear: any data processing task that requires a scalable programming paradigm to be achieved in a reasonable amount of time is at the “Big Data Zone.” If it’s possible, we always prefer to develop with standard tools (like simple R or Python scripting) as it usually results in a shorter and clearer code that requires minimal configuration efforts. But what happens when you start with relatively simple procedures that take several seconds and after three days of developing find yourself waiting more than an hour for one procedure to be ended?

When things get rough, we want to shift our code to the Big Data zone as quickly as possible. In this PoC, we started to develop our solution with standard Python libraries. When we wanted to integrate image processing data into our model, it became clear that we need to shift our solution to the Big Data zone. Since we worked on a very scalable environment, this shift was very easy. We quickly configured our environment so it could run the existing code in a Big Data mode (using PySpark), allowing more complicated procedures to be integrated within it.

Focus on the business problem, not on the dataset

“I have a lot of data. Let’s do Data Science!” may be the first sentence that will initiate an engagement around a data science project. But taking this approach to the rest of the project is likely to result in a burnout. The first two questions that should be asked before initiating every data science customer engagement are, “what is the business problem?” and “can we solve this problem with the available data sources?” Answering these two questions is often complicated. Sometimes it requires conducting a three-to-five-day workshop that includes brainstorming and data analysis sessions with the customer. However, it is a prerequisite for a successful project. Every data source should be considered when working to solve the business problem.

The business goal that the company presented in this PoC was very clear: predict farm yield in terms of tons per acre. Given this business problem, we were able to search for the relevant datasets and available knowledge, whether they were internal from the customer (soil types, previous years’ yields, planted species, etc.) or external (public weather datasets, satellite images of farms).

Pair mode developing

Delivering a complete solution in three weeks leaves no time for distractions and redundancies. In this PoC, we were working as a pair so the risk for duplicated work or disconnected silos (pun unintended: not to be confused with grain silos) was even higher. In order to mitigate this risk, it was necessary to deliberately modulate the work. It was very clear from a very early stage of this engagement that there are two main tasks in the PoC: building the model pipeline and conducting the satellite images processing.

Building a designated model pipeline is a complicated task in which features are engineered from the given data (for example: amount of seasonal rainy days, average temperature in season, previous year yield, etc.) and then digested into a models competition module that choose the best Machine-Learning model for the given problem (e.g. Random Forest, Neural Network, Linear Regression). On the other hand, satellite image outputs serve as a crucial feature in the model, as Normalized Difference Vegetation Index (NDVI) and Normalized Difference Water Index (NDWI) are very indicative for yield prediction. Converting an image into a single number like NDVI and NDWI is a non-trivial task as well, so we decided to separate this task from the rest of the development process and combine its outputs into the developed model when ready.

To summarize, in each of our engagements, we learn different lessons that sharpened our analytics skills. We share these lessons because we feel it is highly important for us to give a true sense of data science work.

Continue reading

Self-Service Eases PC Deployment

Do any of you remember when self-service was reserved for all-you-can-eat buffets? Some of you aren’t old enough to remember when there was no such thing as a self-service gas station. Every gas station used to be entirely full-service, which meant a nice person would come to your car and ask you which grade of gas you wanted and you wouldn’t have to lift a finger.  While the pump was going, they would clean your windows and you drove away with a full tank and truly perfect windows.

In the 1980s, self-service really took hold when people started paying for everything under the sun with credit cards and when credit card processing got so good that you could build it right into the pump it became the norm. Since then, self-service seems to be everywhere and rightly so because it allows buyers to make quick decisions at point-of-sale, and reduces time spent on menial tasks freeing up resources for more strategic efforts.

These great self-service characteristics can even be applied by IT organizations today. For example, Dell IT is now leveraging a newly-launched capability called Shared Connected Configuration to bring the efficiency of self-service to enterprise PC deployments. This new service allows Dell PC customers to connect directly to Dell factories to manage their own custom Windows image—including applications and configuration—and ultimately have their PCs shipped to end users ready for first time login.

Dell IT technicians, who service some 140,000 employees around the world, are putting Connected Configuration to the test and driving efficiencies in our own PC deployment practice.

Taking PC deployment to the next level

In the world of hardware infrastructure, self-service has always meant online support portals where you can search the knowledge base, submit a ticket or post to a community forum.  Connected Configuration, part of Dell’s ProDeploy for PCs, goes beyond the simple online portal by allowing customers to upload PC images, pick their configuration options and keep those selections up-to-date using existing tools.

With Connected Configuration customers connect their Microsoft System Center Configuration Management (SCCM) environment to Dell’s supply chain. That means customers can use the SCCM toolset for image creation to customize and maintain their organization’s Windows image and burn that image on new device orders as part of the factory process. When they order PCs, they can ensure the latest image is installed on the systems in the Dell factory and that they are ready for first time login.

In the past, customers worked with their services program manager to update their PC configurations, shipping the image to Dell’s Customer Factory Integration (CFI) team via a thumb drive or a hard drive. Dell Services would then make sure all of the systems ordered had the specified images.

Besides being able to maintain their PC images themselves, the new self-service option offers another big advantage for users. Before Connected Configuration, an IT technician at the organization receiving the PC would have to perform the second-touch action of adding the new system to that organization’s Active Directory domain before delivering the PC to the user for first login. With Connected Configuration, that step is performed at the factory before the system is shipped.

Using our own technology

Based on our current pilot, Dell IT is very excited about the value that Connected Configuration can deliver to our organization by eliminating the need for our IT techs to prepare new PCs for login across the company. This new self-service capability will also expedite employee PC orders because it will allow Dell to ship the systems directly to the end users.

In fact, pilot results indicate that the self-service process could cut shipping time for a newly ordered PC in half, from 6 to 10 days to 3 to 5 days and totally eliminate the need for IT to do 10 to 15 minutes of onsite preparation per system.

Considering that Dell IT is on track to refresh some 29,000 systems a year for both legacy companies in the U.S. and EMEA, the potential value of this self-service innovation is significant.

Some lessons learned

Enterprise integration with the Dell factory requires anticipating and addressing networking, firewall and other technical challenges, but the rewards can be considerable. Be proactive and consider the following lessons we learned.

1. There’s a huge opportunity to enhance the way PCs are distributed. If you are doing it the old way, there’s definitely a better way to do it.

2. Engage your security teams early and have them be a part of the process in this security-centric integration. You’ve got to have those guys on board and they’ve got to be part of the conversation.

3. This capability has the potential to free up substantial IT resources that can be used to address other value-added IT challenges.

We think this solution is great now and its capabilities are only going to get better for both end users and for IT.

Check out Jim Roth, SVP, Services IT, Dell, at Dell EMC World 2017 in the Services Booth to discuss how Dell IT is consuming services for predictive and proactive support.

Continue reading