I still have a hard time believing this. My view on Google had a near 180 degree shift this morning, after coming across an unpopular tweet calling out Google for trying to patent batch normalization. Batch normalization is a stupidly simple but effective technique to standardized your data being fed through a neural network by its mean and standard deviation.
If your data is
[1, 2, 3, 4]
[0, 3, 5, 7]
[2, 4, 7, 10]
The most basic form of batch norm takes the mean and std of each column
mean: [1, 3, 5, 7]
std: [sqrt(2), sqrt(2), 2sqrt(2), 3sqrt(2)]
…subtracts out the mean
[0, -1, -2, -3]
[-1, 0, 0, 0]
[1, 1, 2, 3]
…and divides by the columns’ stds
[0, -0.71, -0.71, -0.71]
[-0.71, 0, 0, 0]
[0.71, 0.71, 0.71, 0.71]
That’s all. Then it feeds this normalized data to the next neural network layer.
In other words, Google is trying to patent basic math.
Although batch norm isn’t well known outside the deep learning community, it plays an important role as a regularizer, crucial to training very deep neural networks. Google likely did this knowing they are one of the few companies who have the servers and data centers to train extremely 100+ layer-deep networks to approximate say, your future ad preferences. They’ll rather be the only one. For PR purposes, they open-source all their frameworks and offer many enterprise solutions like Google Cloud, but the reality is AI is almost entirely data-driven, and being the only one able to train deeper and deeper networks to make use of all the data will help them monopolize the industry.
https://patents.google.com/patent/JP2019071080A/en (I had to use Yahoo to find this)
https://community.ibm.com/community/user/datascience/viewdocument/the-implications-of-googles-batch
Personally speaking, I’m not sure what’s more ridiculous – repeated applications by Google to patent this idea or the fact no one is seeing what’s coming. The model I’m working with in my internship uses many batch normalization layers – and that’ll possibly be subject to IP infringement? Give me a break.
表裏一体 