John Deere’s first fully autonomous tractor will be available in a limited rollout later in 2022.
John Kim/CNET
By Shara Tibken
Source: www.cnet.com, January 2022
Fully autonomous tractors promise to change how food is grown in America.
Tractors that steer themselves are nothing new to Minnesota farmer Doug Nimz. But then four years ago, John Deere brought a whole new kind of machine to his 2,000-acre corn and soybean farm. That tractor could not only steer itself but also didn’t even need a farmer in the cab to operate it.
It turns out the 44,000-pound machine was John Deere’s first fully autonomous tractor, and Nimz was one of the first people in the world to try it out. His farm served as a testing ground that allowed John Deere’s engineers to make continuous changes and improvements over the last few years. On Tuesday, the rest of the world got to see the finished tractor as the centerpiece of the company’s CES 2022 press conference.
“It takes a while to get comfortable because … first of all, you’re just kind of amazed just watching it,” said Nimz, who on a windy October afternoon described himself as “very, very interested” but also a “little suspicious” of autonomous technology before using John Deere’s machine on his farm. “When I actually saw it drive … I said, ‘Well, goll, this is really going to happen. This really will work.'”
The emergence of fully autonomous farming equipment offers a twist to the century-old contest between man and machine. The rise of automation in everything from car factories to Amazon Go convenience stores has sparked fears that robots will eventually wipe out millions of jobs. But in farm country, where workers are in short supply and younger people are moving to cities, autonomy may be the only way to ensure enough food is grown to feed the world.
Rather than suspicious, Nimz is now eager for autonomy to come to essentially all parts of farm operations. “It’s just going to make my life a lot easier,” he said.
I got an early look at the tractor in action in October in Blue Earth, Minnesota — and even rode along as the machine tilled Nimz’s field.
The tractor is a “way to get the job done on time, every time and do it at a high level of quality,” Jahmy Hindman, chief technology officer for Deere & Co., said during my visit to Minnesota. “It’s 20 years in the making.”
Riding solo
John Deere isn’t the first agriculture equipment maker to develop an autonomous tractor. But as the world’s No. 2 maker of agricultural equipment, it’s one of the most notable. Its signature green tractors are ubiquitous across farm country, and the company has won over young fans with its apparel and toy tractor lines.
CES 2022 marks John Deere’s fourth year at the Las Vegas tech show, an effort by the company to show off the advanced technology in its machines to a new audience. Rather than creating a brand-new machine, the company unveiled equipment that can be added to its popular 8R 410 tractors for full autonomy. Two boxes — one on the front and the other in the back — contain a total of 12 stereo cameras and an Nvidia GPU that let a farmer control the machine from a smartphone, starting it with a swipe of a button and watching live video as the machine moves across a field.
John Deere’s tractors have been capable of steering themselves for two decades — as along as the farmer still sits behind the wheel. That fact makes the move to a fully autonomous tractor less of a stretch for a farmer than for someone who’s spent the last 20 years driving a Honda Civic.
While John Deere may be popular, it also has faced its share of troubles. In November, about 10,000 unionized workers ended a five-week strike that won them better pay and benefits.
One ongoing criticism of John Deere’s tractors is the price, and the new autonomous system won’t come cheap. A regular 8R tractor and the 2430 chisel plow will set a farmer back over $500,000.
John Deere will initially rent out tractors with the autonomous equipment already added, but it hadn’t yet determined the price for adding the autonomous equipment when CNET met with the company. Hindman said it would be “significant” — as much as 10% of the total equipment cost, or as high as $50,000. That may bother some farmers, many of whom are already irritated by John Deere’s refusal to let them repair their own expensive tractors.
At first, the new tractor system will only be able to till fields, which is the process of turning over the earth to return nutrients from recently harvested crops to the soil. Some farmers — pressed for time and lacking workers — skip that process, but that makes them more reliant on chemicals to help their crops grow. By automating the task of tillage, John Deere hopes to take away one responsibility from farmers’ long lists of duties. And with tillage, if something happens to go wrong with the self-driving tractor, a producer has time to fix the problem before spring planting. John Deere views it as the simplest task to automate.
Farmers “are clamoring” for “more and more and more,” said Deanna Kovar, vice president of production and precision ag production systems at John Deere. “They’re just looking to figure out how can they get more jobs done without a human in the seat.”
Not reinventing the wheel
John Deere’s road to autonomy began decades ago. In the mid-1990s, it introduced satellite GPS in tractors to help farmers with precision agriculture, which is the ability to glean insight about what’s going on in the soil in order to become more efficient with gas, fertilizer and seeds. A decade later, Deere’s StarFire GPS receivers became accurate enough to enable tractors to steer themselves.
John Deere started with the 8R because it’s the company’s most popular row crop tractor, and it already “was nearly autonomy ready,” said Joe Liefer, senior product manager of autonomy at John Deere.
“We had the ability already to electronically control the hydraulics and mechanical functions of the tractor, which is required for remote operation,” he said.
The plan this year is to let a limited number of farmers use the autonomous system. During the initial rollout, Deere will rent a full tractor and chisel plow to about 10 to 50 producers who have steady internet connectivity on their farms and have an interest in using the technology.
Later on, Deere will let farmers bring their own tractors to be retrofitted with the autonomous technology. It plans to support at least the past three years of tractors and may eventually support older machines, Hindman said. Because John Deere tractors have had self-driving technology for decades, configuring them to be fully autonomous is relatively easy. It will take only about a day to install the equipment and test a machine before a farmer can take it home to use in the field, Kovar said.
“Retrofitting it on part of the existing fleet of equipment that’s out there … means that more farmers can take advantage of it … sooner,” she said.
The tech inside
Most autonomous cars being tested use a depth sensor called lidar, while Tesla employs an array of cameras, sensors and radar. John Deere, however, believes that stereo cameras are the way to get self-driving technology into as many fields as possible. Its autonomous machine has 12 such cameras.
To make the 8R autonomous, John Deere mounts a stereo camera pod on the front of the tractor and another pod on the back. Each pod has three pairs of ruggedized stereo cameras that essentially work like human eyes. Images are collected by both and are then combined to help the machine locate potential obstacles that are between 45 feet and 90 feet away.
“Just like, say, your iPhone is just taking multiple images and merging them together for … that high dynamic range, we’re using that same type of technology here to really understand contrast and the world around us,” Liefer said.
While stereo cameras work for John Deere’s current autonomous tractor, it may use lidar, radar, ultrasonic and other technologies for more difficult tasks in the future like operating in standing crops, Hindman said. To gain some of those smarts, John Deere acquired a Silicon Valley-based ag tech startup called Bear Flag Robotics for $250 million in August.
“The problem of perception in agriculture is really hard,” Hindman said. “[And] the hardest technical problem for us from a perception perspective is probably standing crop.”
Steady cellular connectivity is a prerequisite for using the autonomous system in a field. If the tractor loses its cell signal, it will stop and won’t start back up until it reestablishes a connection and gets the OK from the farmer through an app.
Inside the machine, John Deere installed multiple ways to stop and manually control the tractor, including through the throttle and normal brakes. The farmer also can grab the steering wheel to take over operation of the machine.
The tractor can still be used manually for other tasks, not just autonomous operations, and the cab looks no different. Because of those changes made to the machine itself, a farmer can’t easily move the autonomous pod from one tractor to another.
Along with the redundant brakes, John Deere has built other security into the system. Data that’s shared between the tractor and the smartphone is encrypted on both ends, and the company embedded security in the software on the machines. The tractor also has redundant systems on board to be sure the cameras and GPS signal are present. If those go offline, the tractor will stop.
Avoiding obstacles
Early on, John Deere’s autonomous technology had troubles with shadows. And leaves. And patches of dirt that were different colors than the others. All of them triggered the tractor to stop and send an alert about obstacles. But the technology is now at the point where the false error rate is low, Hindman said.
To help avoid false alerts of obstructions, John Deere’s autonomous 8R runs with the lights on all the time — an unusual sight in farm country.
Leaving the lights on all the time “makes the environment that we’re taking images of more consistent throughout the day and over time,” Hindman said. And the tractor actually performs better at night than during the day because there’s more contrast in the image data, he said.
John Deere has a US-based tele-support staff to examine obstacle alerts from the tractor. If the workers determine it’s something minor like leaves that blew across the path, they’ll let the machine resume operations without bothering the farmer.
If the tractor comes across something like a large rock, it will stop because the rock could get stuck in the tillage tool. The tractor sends a push notification to the farmer with an image of the obstacle. John Deere built in the ability for the tractor to route around those obstacles so the farmer doesn’t have to come to the field for every issue.
“To make sure that we’re not a nuisance to the customer, we will have support 24/7 that’s saying, ‘OK, the machine stopped and it saw something,'” Kovar said. “Is it safe for us to get started again?”
Real-world testing
Nimz has been testing John Deere’s autonomous equipment on his farm for four years, but he still drives by the field where the tractor is running by itself, just to marvel at the fact it’s actually working without anyone in the cab.
He had jumped out of his combine to talk with me before quickly resuming work. A storm was expected the next day, which would prevent him from continuing the harvest. Farmers aim to get as much work done as possible before rain since they don’t know when they can get back into the field.
Although Nimz has to quit working when it gets too dark, his autonomous 8R doesn’t have the same problem. Instead, it can run all night, getting the tilling done before the dirt turns to mud.
“You want to get as much work done as you can, [but] asking an employee to [work late at night after a long day] is hard,” Nimz said. “An autonomous tractor can do this. … It’s no problem. It doesn’t care. So you can turn it loose and just let it work 24 hours a day to get ahead of weather.”
Because of the ability to farm more hours in a day, Nimz expects to be able to buy smaller equipment. The trend in farming has been to use ever-bigger equipment and tractors to cover as much ground as possible in the shortest amount of time. A 36-row planter can put more seeds in the ground at one time than a 12-row planter. But buying smaller equipment lets farmers like Nimz save money and is also better for the ground. Because the machines are lighter, they compact the soil less.
More than anything, he expects his autonomous tractor and plow — as well as future John Deere products — to make his life “a lot easier.” If the machine is running and he doesn’t have other work to do, he’ll be able to take a lunch break instead of eating a sandwich in his tractor.
“Little things like that mean something,” Nimz said. “It’s things that … I thought I was gonna have to … retire to enjoy.”
A labor shortage
The biggest benefit of autonomous tractors is letting farmers perform two jobs at once. The industry has a labor shortage that makes it difficult for growers to get everything done on their farms in a short window. If a farmer doesn’t get planting done in time, he won’t have a crop to harvest in the fall. If he doesn’t get the crops picked before a freeze and snow, the harvest will be lost.
Farm work is seasonal, with extra workers needed to put in long hours during planting and harvest. In 2019, more than half of all hired farmworkers — roughly 450,000 people — were immigrants, according to New American Economy. But US immigration policy makes it difficult to find enough workers, and the H-2A agricultural visa is expensive for many US growers, the think tank said.
“Farmers frequently worry about finding enough workers as few Americans seem willing to take on the most difficult and physical farm jobs,” New American Economy said.
Most of the population lives in cities, and farmers in the US are getting older, with the average age nearing 60. Farmers will have to feed 10 billion people in 2050 with fewer laborers and without more arable land, Hindman said.
Autonomy could help fill that gap. Automating more tasks on a farm could eliminate jobs that were once filled by people. Nimz, for one, said that when one of his workers retires, he likely won’t hire someone new.
“The guy that’s been with me for 36 years, I don’t have to replace,” he said. “The tractor will take his place.”
But John Deere expects most farmers to deploy those workers to other tasks on the farm, rather than cutting employees.
Autonomy “is not a threat,” Hindman said. “It’s really [the way] we’re going to solve the labor shortage to continue to grow the food that the world needs.”
Swipe to drive
A key part of the autonomy process is the smartphone. John Deere designed the system to be controlled by iPhones and Android phones through its existing Operation Center app. For farmers with the new self-driving systems, an additional tab will show up that contains the autonomy features.
Inside the app, farmers swipe to start the vehicle and press a button to pause the operations. The app tells them how much fuel is left, how long the machine has been running and how long it’s been idle. They can see the obstacles under an “Events” section and bring up — and zoom in on — video from those 12 stereo cameras.
One task the tractor can’t handle on its own is refueling, which needs to happen about every eight hours. A “Go To” button in the Operation Center app can prompt the tractor to head to a preset destination in the field so it can be refueled, much like a robot vacuum will return to its charging station.
While growers can control a tractor from their phones and restart it after an obstacle is identified, they must be physically with the machine the first time they start it for the day.
During the initial setup, the farmer has to map a boundary around the field and around barriers, similar to setting up a VR headset. That can be done by manually driving the tractor or using a John Deere Gator (a small, all-terrain utility vehicle) that’s connected to the company’s mapping technology.
Always-on support
Today, John Deere doesn’t charge service fees for its GPS capabilities or other features inside its machines. But it’s not sure whether it will do so for autonomy. And the company hasn’t yet determined how it will handle upgrades and other logistics. Farmers hold onto their tractors for decades. But the technology used to make machines autonomous will be outdated far sooner.
“We have to come up with the right business model that allows growers to take advantage of increasing levels of technology, increasing improvements in the ability of the cameras and the compute and the perception system without having to change the tractor out every time they want to do that,” Hindman said.
In the early years of autonomy, John Deere expects the updates and changes to come very quickly. While it’s starting with tillage, it plans “to offer new models with new crop types, with new pieces of equipment, with new implements for the tractor … on a fairly fast cadence,” Hindman said.
John Deere says that technicians at its dealerships will be able to fix complicated problems with the autonomous systems. And farmers will be able to make some repairs on their own, like wiring the harnesses holding the pods to the tractor or fixing the brackets that hold the sensors.
Beyond tillage, John Deere will look at other tasks carried out by tractors, like spring tillage and cultivating. And in the future, John Deere plans to extend autonomy to essentially every task a machine performs on a farm, from planting to harvesting.
“Those won’t all come all at the same time,” Hindman said. “So there will be part of the operation on the farm that a farmer still does … in the cab of the tractor, operating the tractor as they do today.”
Farmers will continue to use the 8R as a regular tractor, and future autonomous updates can be rolled out to the same machine through hardware and software additions. That will let farmers use the tractor for more than one autonomous task.
Getting to the point where all parts of agriculture are automated likely will happen “sooner than anybody thinks,” Hindman said. “If my back were against the wall, I’d tell you in 10 years or less we’ll see autonomy all the way through the production system.”
For now, John Deere’s machines will help farmers ready their fields for the next planting season — and I get to see what it’s like from inside the cab of the self-driving tractor.
Growing up on a farm in rural Iowa, I’ve ridden in combines with my dad dozens of times and have checked out John Deere machines that steer themselves. But the experience of sitting behind the wheel of a massive machine as it completely drives itself can only be described in one word: wild.
I’m just along for the ride. Every move of the machine is automated and monitored from a smartphone, letting Nimz harvest corn while the tractor tills his fields. As the machine nears the end of a “pass,” one stretch down the field, I almost forget that I’m sitting behind the wheel.
When I finally exit the machine on the edge of the field, the empty tractor continues on its way, pulling the plow behind it. Rain is about to start falling, and it’s got a lot of work to do.
