About how VKontakte collects information about us. Part 1

[DronVR]

About how VKontakte collects information about us, Part 1.
(C) Vladislav Veljuga

Today, picking the sniffed traffic of the official VK application for Android, trying to find the features by which the API filters out official applications for receiving music, I came across requests of quite interesting content ...

Disclaimer
I want to note right away that I rented You will be liable for any damage that you may cause to yourself / your devices or other / other devices by performing the actions described here. I also not urge engage in similar activities on their networks. Information provided for informational purposes only and extremely not recommended for playback / repeat on personal devices, networks, and users are not strictly recommended , since sniffing someone else’s data is illegal, and I’ll have the right to monitor my device.

And yes, for fans to find advertising where it does not exist: this post / article is not an advertisement. The applications mentioned below are provided as examples, no more.

Update 6
And let's go right away, that's what Andrei Rogozov answered (somewhere) about this information.

Foreword
About two years ago, I also sniffed traffic using Shark for root sent by VKontakte from the phone. I didn’t see anything strange then. Now, when we are surrounded by “smart” (it’s in quotation marks, because they are idiotic) tapes, machine learning, and so on, the technique began, to put it mildly, to follow. On the one hand, it’s good (we give food for machines to be trained), on the other hand, it’s bad (data about us is stored on servers).

results
Having scanned only authorization, audio recordings and generally the first minutes after authorization in the application, you can already catch all these strange requests. So:

The strangest thing that seemed to me was that the application merges absolutely the entire list of user applications installed on the device. What for?! (in the center of the screen slapped the decoded-line of the apps parameter)

Quite often flashed requests to a certain service vigo.ru. The service positions itself as an analyst in the transmission, search for errors, problems and video processing. But it’s strange, because I just logged in, switched to audio and flipped through my wall, where there was not a single video (which should have been automatically played?), And about 5-7 requests accumulated. In addition to notify, there was also network_status.

Here again. wallGetWrapNew - by name it’s clear that this is a request to get someone’s wall (user or community). Why then transmit information about the device? The maximum that comes to mind is for statistics. Well, what is the type of network data for? Also, what doesn’t apply to information leakage: it’s rather annoying that people everywhere are trying to stick in ads - just look at the fields parameter.

The application sends all user actions: went through the menu to the "audio" section, "geo_data" - probably would send an approximate (or even exact) location of the device. The question is, why, if the user does not ask to find something in the vicinity? No, to make life easier for the user by prompting relevant data in the search, for example - there is nothing wrong with that, but why send geodata just like that when opening a post? Unclear.

The application makes benchmarks and for some reason transfers the request time to the API and the download time of the images. Apparently averaged data.

I still do not understand when, under what conditions and why, the invisible WebView is loaded in the application with the page m.vk.com/counters.php. For then two metric pixels (at the bottom of the HTML code) and favicon are loaded from it. Cheat statistics? upd: after another couple of minutes, another request went here, already with different numbers in the img tags - so that the same pixel would not load from the cache, and the request for the metric was still real.

A bit not negative: the application sends error messages if, for example, there was an attempt to upload an image (for example, it was attached to a post), but some kind of error occurred. An example is provided on the screen when the image was simply not on the server (error 404 Not Found). upd: although there was a moment when state = success and no other "identification marks".

Video is even worse. Information about such events as "volume_on", "volume_off" (apparently, turning on / off the sound, but this is inaccurate), "fullscreen_on", "fullscreen_off" (transition and exit to / from full screen mode), the event "video_play", is transmitted which simply sends the current video viewing position, somewhere with a frequency of 10-20 seconds. upd: although Andrei suggested the idea of why this was done: in order to remember the place where the user paused while watching the video, so that he could switch from mobile to PC and continue to look at the PC from where he was last on mobile.

When closing a page (activity) with video, the application asks for the video.viewSegments method, in the parameters of which the ratings (segments) of the timecode that were viewed by the user are transmitted.

All this is the official app. At the time of this writing (July 29, 2017) was version 4.12.1.

In Kate Mobile, such drains were not noticed. The only thing, after putting into operation a new algorithm for issuing audio recordings, both Kate and the official application need to contact Google Accounts to obtain a certain receipt token. And that’s it.

About how applications work on iOS, Windows Phone I can only guess. I did not intercept their packets, and I do not have devices.

I repeat, data such as the nearest Wi-Fi access points, the user's current location, as well as all his actions are not sent by third-party applications such as Kate Mobile, VK Coffee (modification of the official, with cut out metrics, etc.), my client site APIdog, etc.

Update 2
Development friend Andrei Added more screenshots of what merges with the official application for Android.

The name of the access point to which the device is connected, as well as others that are in the reach zone, their signal in dB, MAC addresses.

Plus from it, here that sends the official application for Windows

Only the system version, application version, input method.
Update 3

Eduard Bezmenov , the developer of the modification of the official VK Coffee app, commented on this post like this:

Grigory Klyushnikov , the former developer of this very application, as it turns out, was himself against including Vigo services in the application:

“Vigo has been around a long time, I introduced it. I was opposed from the very beginning, but in my opinion everyone knew how. And it’s you who haven’t even reached MailTracker, the pulp is there. ”
And here is what Vigo actually represents as described by Gregory:

“According to the original plan, or rather, as everyone explained to me,“ choose the quality of the video depending on the Internet. ” Naturally, I absolutely did not want to merge such data for the sake of this useless trifle. We would like to actually make a convenient viewing of videos with fine-tuning quality for the Internet - we would file DASH / HLS, like in real video hosting sites. ... "
Sending a location, as it turned out, is done only when viewing a single post. This does not affect the audio recordings, as some began to believe that depending on the region, some tracks are “hidden”.

Update 5: Responses from VK
Mobile tech support
Denis decided to still get answers to our questions and asked them mobile support VK (id333)

It turns out that your location, data for targeted advertising, a list of installed applications and Wi-Fi networks are vital for the application and the site as a whole.

In response to the last question, support decided to move away from the topic.

About how it was received
Tools at hand
We have a computer for Linux (Ubuntu 16.04 LTS), two phones on Android 5.1 (Sony Xperia L) and 6.0.1 (Samsung * some kind of *). Sony cut Google Play Services. On both phones, the latest version of the application and a third-party application are Kate Mobile (versions 37 and 41, respectively). Well, and, of course, a single local area network, to which both a computer and two devices are connected.

Preparation: Creating an SSL Certificate
If you just start sniffing, we will get only raw data that cannot be processed, since they are transmitted via the HTTPS protocol - the data is encrypted. In order to obtain data that we can decrypt, we change the certificate to the one that we created. This way we can decrypt this traffic.

 openssl req -x509 -newkey rsa: 4096 -keyout key.pem -out cert.pem -days 365

This command creates a certificate, where key.pem is the key file, cert.pem is the certificate.

In the phrase key, enter something like a password. We still need it. Then repeat it again. The remaining fields can be left blank / not entered. At the end, two files will be created in the current directory.

Preparation: installing our certificate on the device
We transfer the cert.pem file to the device and install it on the system. I’ll draw attention to the fact that in order to install a certificate it is necessary that the phone has some kind of protection on the lock screen (graphic key, password or PIN).

Step-by-step certificate installation on Android 5.1

The certificate is installed.

Preparation: Port Forwarding
We return to Linux, drive into the terminal:

 sudo sysctl -w net.ipv4.ip_forward = 1 
 sudo iptables -t nat -F 
 sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080 
 sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8443 
 sudo iptables -t nat -A PREROUTING -p tcp --dport 587 -j REDIRECT --to-ports 8443 
 sudo iptables -t nat -A PREROUTING -p tcp --dport 465 -j REDIRECT --to-ports 8443 
 sudo iptables -t nat -A PREROUTING -p tcp --dport 993 -j REDIRECT --to-ports 8443 
 sudo iptables -t nat -A PREROUTING -p tcp --dport 5222 -j REDIRECT --to-ports 8080

Preparation: Ettercap
Install Ettercap:

 sudo apt-get -y install ettercap

After installation, we launch it.

Data sniffing
Click "Sniff" -> "Unifed sniffing ...". In the interface selection window, the one you usually need is usually selected (maybe wlan0, wlp1s0, enp5s0), if not the one, choose your own. "OK".

Interface selection
Next: "Hosts" -> "Scan for hosts". We are waiting for host scans.

Host Scan

Further "Hosts" -> "Hosts list". In the list, select the IP of our router (I have 192.168.1.1) and click "Add to target 1", then select the IP devices (I have 192.168.1.222), then "Add to target 2".

192.168.1.1 - router - target 1; 192.168.1.222 - phone - target 2

Next, "Mitm" (Man in the Middle) -> "ARP Poisoning" -> set the flag "Sniff remote connections" -> "OK".

Next, "Start" -> "Start sniffing".

End of preparation: SSLSplit
Next in the terminal we set sslsplit:

 sudo apt-get -y install sslplit

When the installation is complete, create the directories:

 mkdir logs

And in the current directory (where the cert.pem and key.pem files are located)

 cp / dev / null logfile.log

We exit the account in the application on the phone.

In the current directory, execute:

 sudo sslsplit -k key.pem -c cert.pem -l logfile.log -j logs -S. https 0.0.0.0 8443 http 0.0.0.0 8080

Enter the phrase key that was specified when creating the certificate.

In logfile.log incomplete logs will be recorded (namely, domain, address, port), detailed requests, headers and answers will be written to the logs directory.

Next, log in to the application and see how data appears in the terminal, in logfile.log and in the logs directory. To stop sniffing, press Ctrl + C in the terminal.

Logs in the logs directory will be written under the owner and root group without access to read and write from the current user. Therefore, you need to change the owner. In the directory with certificates, enter

 sudo chown -R vlad805: vlad805 logs

Where instead of "vlad805" is your username.

Next, you can view files using a regular text editor.

Update
As Anton later suggested, sniffing can be done with two clicks using Android applications, and then this long instruction will not be needed. But ... to whom it is more convenient.