🚀 Add to Chrome – It’s Free - YouTube Summarizer
Category: Data Science
Tags: agriculturedataenvironmentmachine learningsatellites
Entities: BarcelonaCairoCopernicusGambiaLandsatMoroccoMoscowPeteSabadellSentinelSpainVenice
00:00
[Music] as Pete was saying I have a several too many years working with data in a broad
00:15
range of data I started actually working with industrial manufacturing data steel production and then transportation data city data telco data and quite a bit with financial data and it was from each
00:32
one of the changes it was fascinating diving into the data and getting getting your heart hands dirty which I guess is what the community here likes to hear about a data but I want to start explaining some of the shortcomings of
00:49
all the data that I worked with and basically a though this data can solve a number of very relevant challenge challenges on each one of these respective industries they fall pretty
01:04
short in solving some of the major challenges that humanity has today so if you look at humanity as a whole people globally and I'm going to go through some very very basic examples you know if you look at food this is the number
01:21
of land that has been that is being used year-over-year for food production in order to feed humankind as you know we're being very successful in food production not so much in food distribution but we
01:40
are producing more food than humanity needs but this is at a high cost in terms of environment so if we want to continue increasing food production by 2050 there will be about between 9,000 10,000 of us here will need to double
01:56
food production from today to that year because we need to feed the people but we also need to feed the animals so if we want to do that in a sustainable way we basically have three options either we go in plant in the Sahara or
02:12
we DeForest the Amazon basin or we find new alternative seats in ways of a more advanced ways genetically of producing foods or we do what we're doing more
02:29
efficiently we use the land we are already using more efficiently and there's a lot of evidence to show that this last option is preferred over the first options and data is very relevant in order to achieve it let's look at
02:46
another challenge here this is global primary energy consumption so as you can see not only it's going exponentially like a most of the indicators that that we're showing but it is also growing
03:02
along some of the lines we're not so interested in growing like cold and crude oil and natural gas because there's like finite resources so we need ways of managing much better not only how that grows in order to use energy
03:19
more efficiently but also in the distribution between different energy sources so this is a huge challenge that we have ahead not only at the level of the production but also at the level of the global distribution of the energy another huge challenge that we're
03:37
struggling with is the management of natural resources so what you see there that's that's that graph shows a mixture of forest gain for its loss but you have loss of forest is inbred very worrying
03:52
along the equator which is where most of the canopy is concentrated but you can see on the other side that you have net gain of forest in many of the developing countries so there's a lot that there's some positive indicators that if we
04:10
manage to to to arrange the production and the use of land more efficiently related with energy related with food we can regain a forest production and we need to manage this the thing is how do we manage this and kind of data what is the solution for
04:27
this so the the the the what I we know a couple of things that whatever solution to these huge problems there are a it has to be global right we cannot have a solution that's good in Spain but then
04:43
it doesn't work we're gonna have food solution that is good at Spain that doesn't work for the rest of Europe we can't have regulations that finish in any border because we know that in the globalized world we have anybody will be able to bypass the the regulation but just jumping across the border we need
05:01
to we need to have that solution to be global so we need to look at it globally in a way what we need and we don't know which is the right solution for each one of those but it but at the very least what we want to know is whether one solution is better than the other one as we don't doubt that creativity human
05:18
creativity will come up with many possible solutions for all these challenges I don't doubt about that but what I'm not sure is how are we going to cryed which ones are good and which ones are not so good so in a way inside a
05:34
logic what we said is we need something like a Fitbit for the planet that at the very least we're measuring each one of these solutions each one of these policies as they are taking each one of the changes and were able to tell this
05:49
is good this is bad in another to do that we need to do it at a planetary scale and there's many ways of doing this we could we could send aerostatic globes we could cover the planet with drones we could have earth sensors every
06:05
10 meters to measure all this we could empower our telephones to do more things that they're doing but a that's not at a planetary scale even if we have our cell phones the land cover of the earth by humans is very limited and a lot of
06:22
these things food production energy production are happening where we are not right in remote places so the reason SATA logic was never intended to be a satellite person it was a company it was intended to be a
06:37
solutions company for these kind of challenges turns out that satellites are just great as remote sensors for acting as a Fitbit of the planet the challenge then was that satellites building a satellite that took 1 meter resolution
06:54
imagery of the planet ten years ago cost 650 million dollars okay so the cost of opportunity of having each one of those images and the coverage of those images was just too limited in order to have daily let's say
07:11
coverage of the planet at 1 meter resolution you need hundreds of those satellites so it simply does not scale so we need to make satellites much cheaper much lighter much smaller okay because cost also is related with the
07:28
launching cost and weight has a lot to do with that and if we manage to do that we could scale that coverage we could have images of the planet every single day every single week and most importantly if I manage to do a
07:44
satellite for this in a million and I'm talking a thousand times improvement from the previous version I reduce my learning cycle and as you know with technology the key is reducing your learning cycle if you manage to do
08:00
anything any kind of experimental environment anything which allows you to learn faster then you enter into a cycle that you really can do breakthrough stuff right and you need to do all that that reduction while preserving data quality and what I mean by data quality
08:16
here is taking good 1 meter resolution images because why 1 meter why not 10 meters for instance right now you have free imagery of the world every 5 days thanks to the Copernicus to Sentinel satellites to the Copernicus constellation or to Landsat data a
08:33
little bit lower resolution a but it has 10 meters 15 meters and the reason for 1 meter is 1 meter is the scale of human beings it's our scale we we live at 1 meter scale you know our roads are this size
08:48
because we are this size and our cars are this size because we are this size in our machinery and buildings and everything we do our impact in the planet is in the meter scale if we were one centimeter in size probably would lead we would need much higher resolution satellites than
09:04
we than we do now so we did that and we currently have a eight satellites in space and these are sample this is sample image of here of Barcelona at 1 meter resolution taking bar satellites earlier earlier this year
09:21
and you can do Barcelona which is heavily populated but on the other side it's also heavily populated with sensors so the value of doing Barcelona or any other large city is not so important and I'm sure that many of the data that most of you guys work here be it transportation uber lyft a kavithai a
09:39
global financial data can give a very good image of what's going on in Barcelona but not so good in in in West Liberty Iowa right a where a lot of the production of a in this case corn and
09:55
soy is taking place mostly production in the United States that's the call the Corn Belt in the United States and on top of that what we did is in not only he added a 1 meter resolution camera what we did what's called a hyperspectral 30 meter resolution camera so hyperspectral and I'm mentioning this
10:12
because I'm going to show you examples on computer vision on each one of those sensors so you need to understand what they do a hyper spectrum means that basically our hyperspectral camera looks over the visible scale plus the near-infrared but instead of having three different channels like we would
10:32
have it has 30 different channels that looks into very narrow bands in a way it's like have it looking at colors with superpowers like where we see maybe the same plant being having a green leaf a hyperspectral camera would look that
10:48
that could be a a million shades of green and thanks to those shades you would know the health of the plant so you can do a ton of nice things this is Gambia here for instance here in Gambia we're looking at the presence of algae in the
11:04
rivers and the reason we're looking at the presence of alga is because the where those alga I live lives some tiny snails and those snails are where these parasites that giving Gambians huge
11:21
headaches because they're causing gastro problems in children inhabit in these are parasites that inhabit in these snails they're transported by these snails so by looking at the alga you can anticipate where this disease this
11:39
parasite disease will be spreading because where you have huge growth of alga you will have a higher probability of that disease spreading in nearby towns so so you already can see through a very simple example looking this how
11:57
this new source of data can help address some challenging problems and here just just a curiosity example is we not only do the images we also do video is that not not so good video but what you're seeing there all the time is Venice and
12:13
what I like that's Venice you know the islands around Venice and so forth and it was a first video we got so we're proud of it not because of its quality but because of its originality and what's really interesting about this video it gives you an idea of how fast these satellites move in order to
12:29
capture the data the video as you've seen it started shooting when the satellite was over Moscow in it finished shooting when the satellite was over Cairo so when you have these two sources of data the multispectral data and the
12:45
hyperspectral data you can address as I said challenges along many different a sectors a government sectors agriculture infrastructure in many different ways and what I'm going to show you one is very concrete examples of the basis of
13:02
those challenges because you need to build a solution for instance for people whom I'll manage or remote very expensive oil infrastructure it's really expensive too people their airplanes drones so doing it remotely will save them a lot of
13:18
money but in doing that solution you need to talk their language you need to do something that will integrate with their oilfield management systems and so on but what underlies always these systems right between the data and their solutions is computer vision you know
13:35
machine learning computer vision image processing so these are the four and you will see nothing nothing new here not meaning that everything is solved here this is the four main a let's say algorithm types or algorithm groups
13:51
within computer vision that are relevant in remote sensing segmentation detection change detection meaning detection plus looking things over time and regression okay and this is what you're seeing
14:06
there is an actual image of one of the projects we have this is a large four station right a in a forest operation and you're using segmentation to determine where's the actual forest worse the actual trees and you can see
14:22
those blue lines there a versus where there's nothing you're using detection to actually detect the individual trees and count them we're using change detection to look at the evolution of play of harvesting of the different
14:37
activities that take place within a four station and we're using regression to estimate from the time series of those trees growing the potential value of all the four stations single problem we're using all those algorithms there let's
14:52
see some some some more detailed problems so segmentation this is the classical one this is the one that probably a it's been around for a while since Landsat started sharing their images a remote sensing community and some computer scientists started looking
15:09
at how to do land-use basically it was done an image processing huge jump when machine learning came up and in the last couple of years as we're having more data available for companies like us but a lot of companies that are following the same
15:25
goal you start there's that there's a boom of this last cvpr there was a huge challenge on on for instance on land use which has been a traditional problem for a while so you can see here this is Sabadell actually this is here I try to take pictures a
15:40
near Barcelona so and this is a typical problem of land use which is assigning a pixel to a type of use or to a type of cover that land is having be it different types of forest agriculture residential buildings industrial
15:57
buildings and so forth only doing that you can do a lot of interesting stuff segmentation another one here so this is another problem we are this is an in this case this is called rapeseed which is the the crop that is used for canola
16:13
oil which is a very it's growing a lot it's quite good because it's reclaiming palm oil which is not so good right in it so it's growing in South America it's growing in China it's growing mainly in Canada it's beautiful because it has all these yellow flowers so it's important
16:30
because there's since it's growing so much the growth of this by the individual farmers is not the precisely monitored in many ways and that needs to be monitored to understand what the impact of that growth will have on the market so we're detecting automatically
16:46
canola oil from plantations and this is a very simple it's simple it's not so simple but it's a solvable a segmentation problems and you can do more stuff here what we're doing is training a neural network what we would call semantic segmentation instead of doing it by a simple 0 1 training here
17:04
we're training the a neural network for instance on roads or on rivers so you have this image here and we want to isolate buildings and different types of of crops and as I said land use you know
17:21
roads and rivers and that's the risk what you're seeing there directly is the response of the train network on roads here and on rivers over there and you do more also more more traditional
17:38
detection in this case we're detecting boats in a lake so this is a recreational area national parks that want to assess the human use of that National Park so they can control when things are going out of hand when
17:54
there's too many people they can restrict the entrance and they can restrict the usage of the lakes in those national and recreational parks this looks like it looks all of these look straightforward tons of challenges within each one of these tons of
18:09
challenges to be done day so just to give you an example here we're talking about a 15 percent error rate that could be easily taken to 10-5 given more data but really if you want applications on
18:24
this you need to do even better than that I was mentioning energy production of energy here what you have is an oil field and oil fields need to be looked for many different directions typically the oil field belongs to the
18:40
government and it is a given to different companies for exploitation so the government wants to know that that is done in the right way for instance from an environmental purpose and each one of those companies want to understand each one's operations and
18:57
they also want to understand the other guy's operations right so there's a lot of information that you can extract and this is actually you know the response of things like I showed you that nuraghe that road train neural network in Pat in a paddy tech chure and an oil pipe
19:13
detection on this particular image that you see here that's the response there and that's the actual a ground truth that we're obtaining as you can imagine and I'm going to finish with some of the challenges or top challenges we have but
19:28
as you can imagine the difficult chroma like we say here there is the ground truth the difficult thing to obtain is the ground truth so now if you add to detection you that time you have this kind of situation this is exactly the same region right and you can say
19:45
okay do a change detection algorithm if anybody you know for those who here who are aware of image processing in computer vision a what you have there in the middle that's an oil spill and that's exactly what you want to detect
20:00
that's what your system you want to take one day there's an oil spill you're looking at the planet and you know there's an oil spill there and you can act upon it immediately the problem is that any traditional change detection of two images will put will say that those
20:16
images are essentially if different on every single pixel if you look at that so our challenge was to train detectors in order to look at at these these are exactly once again these two images that you're seeing horizontally are exactly
20:31
the same images and the regions that you see surrounded by red is where we've seen changes relevant changes so you would say that semantic change detection in the sins I don't want to look for vegetation changes I don't want to look for seasonal changes because there's a
20:48
draught or because it rained right before an image or any other type I just want to look where there's new in this case oil pads or your pools roads in the other case where there's a new oil spill or in this case I want to look where
21:03
there's harvest there has been harvesting in forestry and where there has been deforestation in large forest this one is a nice one because here what you're doing is you're training a classifier on solar panels and this is
21:19
related to understanding energy right at a global scale you're training on a solar panel and you're actually looking at the image and the response let's say to a city of a city or a certain region to the solar panel detector at two
21:37
different times and by subtracting those you can have those heat maps right away telling you where there were solar panels installed in that lapse of time and you can do it there's many more examples but I kind of feel attached to
21:53
these examples because every time I look at the images those is exactly the same into the same place and you see that the most relevant changes are around even shadows are a problem here because the satellites not always take the image or capture the image from the same angle
22:09
it's a very tough problem what you actually want to say is only show me only the changes in the building or show me only where the growth of the river might pose a problem to a certain type of infrastructure and that's exactly what these detectors are finally in
22:27
finally regression a I said segmentation detection change detection and regression you see here in this particular case we are in a merging fusing very different data sources to
22:43
estimate soil in rice plant a soil moisture in rice plantation so soil moisture is not so easy to detect because the plant covers the soil in many cases so you have to look through the plants so what we're doing is not
22:59
only using the visible channels but also using near-infrared thermal infrared channels to build these to regress the amount of soil moisture and if you have a this is this is the same this is the
23:15
soil moisture estimation the water level indices in the whole plantation and if if you have in this case what you have is for a certain or a rice field the yield that that rice fields had at the
23:32
end of a campaign so a rice the harvesting rice planting campaign can last about 4 months ok so here you have regression plus time where you end up having some kind of predictive analytics so what I want to be able to do is at
23:49
any time of the campaign be able to predict the actual yield that I will have because in agriculture a lot of people move by proxies they're just saying ok I'm looking at work with the amount of hydrogen or the amount of water or fertilizers that I need at the
24:06
end of the day because I want more yield so if we're able to look at the actual yield because we're to capture that information in week after week predicting yield based on current situations every week we can act on our plantation not based on the water
24:24
level that is important but not the only thing but based on the actual yield same thing in this case we applied this thing for a estimating future forest density so with that I showed you I hope you
24:39
know some good examples of computer vision applied to these types of images across many fields and I mentioned that in this case this is a solution that is being developed of transforming these computer vision results into an alert
24:55
engine for operators of oil pipelines and you can have this kind of things across infrastructure as I said Kisuke managed roads and bridges and power plants and so forth energy food security and also natural resources governments
25:11
into cartography policy and government you wouldn't imagine because we're used to Google Maps and we're used to open street maps and those are great were there's people but there's not people you know most of the earth there's no people and they're really bad you
25:28
wouldn't imagine how bad the cartographic status of certain regions of the planet is today and in order to start understanding once again what we need to do with the planet you need to get that right from the beginning so I
25:43
want to finish by especially because this is a technical community to have pealing - the technical skills and basically sharing with you that were far from having this problem solved so I chose a single problem a single sector
26:01
let's say which is the ad tech you know technology applied to agriculture sector and the the kind of Holy Grail in agriculture is what you see there that tractor who's being told directly from
26:16
the satellite the percentages of whatever seeds or fertilizer or water off whatever needs to apply to be applied in order to rationalize the use of resources in order to optimize the
26:34
productivity okay and that's that would allow what I said at the very beginning to have a much better use of current resources without depleting you know the jungle or the desert right but even
26:51
doing things here is very tough and let me let me show you first of all there's a lot of variability between crops you you work for for for three months in order to do this great thing about soy and so much or rice moisture that I show it won't work with wheat right it won't
27:08
it has absolutely nothing to do but then you have it won't even work with with the same a crop in a different region so you have rice in Argentina and you have rice in China whole different story
27:25
different soil so you have to do it all over again think from it as a machine-learning expert the ground truth that you need you need that ground truth all over again in many place and then it doesn't work on the same crop on the
27:41
same land because you have generic variety and I might have this field here that comes out of certain variety of rice in the field next door is a different variety of rice so if I train a model on this field here it just won't work on the field next to its same soil
27:59
same crop then of course it won't work between different campaigns because I have same soil same crop same field next campaign next year first day rains a lot everything's risk different from the
28:15
rest of whatever I inferred on the previous campaign won't work on this one and on top of that you have regulations so across countries you might have that certain types of fertilizers that certain types of seeds are allowed and not so the kind of measures that you can
28:31
prescript in one place or the other will completely different and that's only the variability issue in terms of learning there's also the availability and the typical lack of availability that we will find in many fields is lack of ground truth the lack of digitized data
28:48
and the truth is that in many sectors you know the farmers are still writing them down in notebooks their observations everything they write down but in notebook so there's low levels of digitization and availability in this
29:04
particular field is also low because it's it's learning cycle it takes a year it's not like the Drosophila right like that you can do to your genetic advance was really fast because of the generations this takes a year you know so you need to test whatever you learn
29:20
next year and on top of that this is one you know this is a kind of thing where startups are crashing like bugs against the light you know that you can see attics startups a starting and ending their lives every single day because
29:36
they don't realize these kind of things for instance a the genetic lifespan of a certain crop variety the average is 3 years today so if I'm planting wheat the probability in five years that the seeds
29:52
the genetic variety of the wheat I planted today is similar is is very very low okay so that means that imagine first from a machine-learning perspective first campaign you have no idea it's a new genetic variety so you try to gather all the data you have
30:07
second one you have one data point and the third one you have two data points hopefully you're able to predict something and the next one there's a different genetic variety so there goes all you and on top of that you have new sensors I mentioned that hyperspectral
30:23
sensor it's very exciting a say okay we have hyperspectral data but we don't have hyper spectral ground truth so how do we infer from the ground truth we have that is measured that is captured with other sensors and of course all the things we're doing are very nice and on
30:39
top of all those things we need to provide information to the farmers in a real-time so here's there's many but these are the three most relevant lines of research that are pretty hot today that are very relevant for what I
30:56
do number one is automated data generation okay I don't need to wait for the farmer for the whole year you know I need to be able to simulate all these cycles I need need to be able to simulate the the
31:11
behavior of the genetic varieties I need to be able to simulate and get data even if it's not exactly the data that I need in order to train the models that I need one-shot learning very important because of all the examples that I gave you this
31:27
is kind of self-explanatory and with one that I believe is huge is of course transfer learning and domain adaptation for instance if you look at regions like Europe like in Spain were very advanced in terms of olive trees right and we
31:44
have pretty much a good idea especially in where there's expensive production high-value production we probably have a name and last name for every single olive tree over there right and we have a lot of data points for the olive trees and there's a lot of investment in
32:00
gathering data but then you go to the olive tree production in Morocco and you have no idea okay so maybe this kind of satellite data that you can have cheap or almost free in both Spain and Morocco
32:15
can be a great transfer medium for all the things that you are able to learn with algorithms in Spain and apply in Morocco so domain adaptation for places from MU you know moving from places where you have a lot of ground truth
32:31
data available to places where you don't have ground truth available especially because this typically agrees with places that you have high levels of development to places where you have low levels of development will be huge and there is a very classical machine
32:47
learning and computer vision problem so with these challenges I wanted to finish and invite you to to ask questions and if don't worry we'll take a couple and and we'll go afterwards outside for if there's anybody who wants to chat a
33:03
little bit more about these challenges thank you very much [Applause] thank you very much very interesting could you speak about data volumes and
33:19
data stack to be able to achieve that something that we that you're dealing with I didn't get the last part you mentioned about data stack data stack like are you using a Hadoop cluster to process the data or are you using other
33:35
technologies yes and data volumes are you dealing with a very big data sets or yes how does it work in that specific space yes yes I wanted you know we had a discussion about Peter at the beginning I wanted the presentation here to focus more on the challenges and the work
33:51
that's being done I have another presentation that was presented a while ago and data beers they had more to do about the challenges due to the volumes of data because here we're talking petabytes let's say just the earth at
34:07
one meter resolution is half a bit about the day okay so so there's a lot to be done there the interesting thing is also here that you have different communities that have developed different platforms that are relevant for us of course you have the typical data stacks of the big
34:24
data can be the traditional stacks that you're talking about of the big data community but this is per very particular data because it's geo localized data so there's some things that are easier in the sense for instance if you have multiple data sources it's pretty straightforward how
34:40
you stack them one on top of the other in order to have all the information because they refer to a particular point so there's a lot of work related to that to managing geographic information that has been developed for searching in
34:56
geographic information like post Chris a G's a specialized for geographic information system that has been developed in the G is a community and then you have all the more machine
35:11
learning let's say a tools what we found from the beginning is that we would have to develop and I think that the different companies that are dealing with them with this are pretty much in the same situation with what I've spoken with him we have had to
35:28
develop our own a stack for dealing with this so basically we're working of course on on cloud because this is you you cannot work locally with this kind of information and and then the rest you
35:43
know we're using most of the development we're doing is basically pure Python and on top of that we're using the traditional libraries both for for a handling large amounts of data and for handling geographic information data
36:01
from from from the Python library so we're developing it ourselves we're happy to to show you the the details of the platform if you're interested outside yes
36:26
right you showed this soil moisture modeling which I thought very interesting on it I'm working undirected I heard you before better than now with a micro okay I know so you showed this soil moisture modeling and do you have also modeled smell like penetration of
36:44
this water to lower areas within the soil or a we don't right now there are a lot of models for this a once again a ground truth gets you really far and in
36:59
certain places of the world there's amazing to a lot models of geological formation that go a so you can mix that in order to infer water penetration models there's a lot of there's not too much work done on the there's a lot of work done here from the more physical
37:16
standpoint from the traditional remote sensing or geological the communities not much merging this with new machine learning techniques so there's a bunch of opportunities to do interesting things there from a remote sensing
37:32
perspective a it's very difficult to do this you would need a go to go with sensors that go completely either fully into the infrared or into the ultraviolet and there are these sensors
37:48
but in many cases these sensors are expensive and still not validated in terms of accuracy so we're not using that but but already by merging this with existing geological information there's a lot still to be done
38:12
you