CIOs spend a lot of time and resources on projects related to analytical systems for processing sales information and other standard business data. At the same time, dashboards are created for managers to display company performance indicators and help them make forecasts for the future. Such systems bring significant business benefits, but in fact, the opportunities they open up are only a small part of what can be done with the data available to the organization, says Krishna Nathan, CIO of S&P Global (formerly McGraw Hill Financial), which deals with credit management. ratings, as well as providing consulting and analytical services for the stock market. Under Nathan's leadership, a new enterprise-wide data processing system was designed and implemented, implementing a strategy aimed at accelerating business growth and creating new offerings for customers.

Some companies are starting to collect additional data - they call it alternative, non-traditional or orthogonal. This is still a new direction, but CIOs should become familiar with the relevant technologies today. After all, very soon alternative data will become a mandatory tool for many companies.

However, do not rush to hire yet another expensive specialist. Let's figure out what we're actually talking about.

What is “alternative data”

Nathan's definition of alternative data is that it is data that comes from non-traditional sources and can be analyzed to provide useful insights beyond what you normally get.

Let's say you have a retail chain and you intend to open a new store in another city. Typically, such a decision is based on the performance of your stores in a particular city and other cities.

An alternative source of data here could be photographs of supermarket parking lots taken over several months - parking occupancy levels can be correlated with sales volumes. As well as information about pedestrian traffic in the area where the store is planned to open. By combining the information you receive, you can learn something new that will help you in your business.

S&P Global also provides analytics services to the commodity exchanges, and the CIO has to constantly think about how to offer customers additional insights using alternative data sources, how to combine disparate information to give clients information they couldn't get anywhere else.

Let's say S&P Global has information that an oil refinery in Rotterdam can produce 100 thousand barrels of petroleum products per day. But due to supply shortages, approximately 70 thousand barrels are processed, that is, free capacity for another 30 thousand is available. What happens after an oil tanker with 30 thousand barrels enters the port? “If the plant's available capacity report is from a week ago, we won't know that the oil has just been unloaded,” explains Nathan. – That is, traditional data is outdated. This is where a source of alternative data such as satellite imagery comes in handy. If we analyze satellite imagery along with other sources, we will get a more accurate picture of reserves and production in almost real time.”

Alternative Data and the CIO

Even if you don't have ready-made application scenarios, get acquainted with new technologies. Plan systems that allow you to combine multiple data sources for analysis. Learn to manage the data delivery chain, protect it, and take into account usage rights. And hire the necessary staff - you need experienced data scientists who can analyze data and extract useful information.

To quickly launch a project in the field of alternative data, you can use a ready-made solution. That's what S&P Global did when Platts, a subsidiary of the company, acquired cFlow, a tool for interpreting satellite imagery. CFlow offers visualization tools that allow you to monitor changes in trade flows along vessel routes and provides information on the volume and nature of tanker cargo.

Convince company management that the time has come to invest in alternative data - buying existing solutions or creating your own. Some of your alternative data projects will work, but many will fail. Well, if alternative data brings truly valuable information, use it to receive funds for new projects.

– Martha Heller What is ‘alternative data’ and how can you use it? CIO. JAN 3, 2017

    Most users of modern operating systems of the Windows family have encountered a situation where the help file in CHM (Compiled Help Module) format opens only partially - you can only view the table of contents without the contents of its items:

Additionally, if you try to open a CHM file contained on a network share using a UNC (Universal Naming Convention) path such as \\server\h\help.chm, its sections are not displayed. In other words, you can normally view .chm files only if they were not received over the network.

A similar picture occurs when you try to open an executable file that was downloaded from the network. You will see a security warning:

Moreover, the same file, extracted from an archive that was also downloaded from the Internet, can be opened on this computer without any problems. In fact, the only difference is that the file being opened was created locally, during the unzipping process, and not downloaded over the network. In other words, Windows has the ability to determine the network origin of a file, and respond to it using certain security settings.

A mechanism for determining the network origin of files.

In the NTFS file system, each file (or directory) is represented as a collection of individual elements called attributes. Elements such as the file name, security settings, and even data are all file attributes. Each attribute is identified by an attribute type code and, optionally, an attribute name. So, for example, the file name is contained in the attribute Filename, content - in the attribute DATA, information about the owner and access rights is in the attribute Security Descriptor etc. The contents of each file ($DATA attribute) is a set streams, in which the data is stored. For each file or directory in NTFS, there is at least one main thread in which the data is actually stored. However, in addition to the main thread, a file or directory can also be associated with alternative (A alternate D ata S stream - ADS), which may also contain some data that is in no way related to the data of the main stream. The main stream of the file has no name and is designated as $DATA:"". Alternative streams must have a name, for example - $DATA:"StreamData"- alternative stream with name StreamData

When the functions of writing data to a file are performed, they are placed in the main data stream. When we open, for example, a text file with Notepad, we get access to the data of the main thread. The data of alternative streams, when using standard access, is not displayed, and indeed, there is not even any sign of their presence. However, alternate stream data associated with a specific file or directory can be accessed using special programs or by using special syntax on the Windows command line.

For example, writing text to the file test.txt with the command echo:

echo Main stream Data > test.txt- write the text “Main stream Data” to a file test.txt, which means writing to the main unnamed stream.

But you can change the command:

echo Alternate stream Data > test.txt:stream1- write the text “Alternate stream Data” to an alternative stream with the name stream1 file test.txt

Now you can open, for example, each of the streams with Notepad:

notepad test.txt- the contents of the main stream will open with the text “Main stream Data”

notepad test.txt:stream1- the contents of the alternative stream will open with the text “Alternate stream Data”

Alternative streams, being invisible to standard tools for working with file system objects, are nevertheless very often used to store additional information about files and other service information. So, for example, when downloading files from the Internet, browsers add an alternative stream called Zone.Identifier, which can be opened with notepad, as in the example above

notepad %USERPROFILE%\Downloads\ChromeSetup.exe:Zone.Identifier- open an alternative stream with the name in notepad Zone.Identifier ChromeSetup.exe You don’t have to specify the path to the file by first executing the command to go to the directory of the current user’s downloaded files (with the standard location of user service folders):

cd %USERPROFILE%\Downloads- go to the directory of downloaded files.

notepad ChromeSetup.exe:Zone.Identifier- open an alternative stream with the name Zone.Identifier for the Google Chrome browser installation file named ChromeSetup.exe in the current directory.

As you can see, the contents of the alternative stream contain the lines:

- section sign with a description of the data transmission area
ZoneId=3- zone identifier.

This information makes it possible to determine the origin of the file by identifier number ZoneId:

0 - local computer (Local).
1 - local local network (Intranet)
2 - Trusted Sites
3 - Internet
4 - dangerous sites (Restricted Sites)

This definition of zones, for example, corresponds to the security settings of Internet Explorer:

In this case, you can determine that the file ChromeSetup.exe was obtained from the Internet (zone ID = 3). When running such a file, a security warning about an untrusted source will be issued. The security features of Microsoft Office applications work in a similar way when they warn about the danger of opening files that have been downloaded from the Internet. For the same reason, the contents of help files in the CHM format do not open - the contents of the alternative stream allow them to be classified as dangerous, regardless of the actual or non-existent danger.

Try changing the same notepad, the ZoneId value to 0 , which will correspond to the local origin of the file, and the security warning will disappear, as will problems with opening office documents or help topics in .chm files.

A similar behavior of security systems will occur in cases where the contents of the alternative stream are deleted (make it empty), or the alternative stream itself is deleted altogether.

Starting in Windows 7, you can use the command to get a list of alternative file streams DIR with parameter /R:

dir /r %UserpRofile%\Downloads- display a list of files and alternative streams in a directory Downloads current user.

To work with alternative streams in any version of Windows OS, you can use the utility streams.exe from the Microsoft Sysinternals Suite software package. The package contains many small programs for diagnostics, optimization and administration, including a utility that allows you to make up for shortcomings in working with alternative streams.

Command line format:

streams.exe [-s] [-d]file or directory

Command line options:

-s- process subdirectories.
-d- delete alternative streams.
-nobanner- do not display the start banner and copyright information.

Examples of use:

streams.exe /?- display help on using the program.

streams myfile.txt- display information about file streams myfile.txt

streams –d myfile.txt- remove alternative file streams myfile.txt

streams -d -s D:\Downloads\*.*- remove alternate streams of all files and subdirectories in a directory D:\Downloads\

On Windows 8 and later operating systems, PowerShell also allows you to work with alternative threads:

Get-Item -Path -Path C:\FirefoxSetup.exe -Stream *- display information about threads in the file C:\FirefoxSetup.exe.

Get-Content -Path C:\FirefoxSetup.exe -Stream Zone.Identifier- display the contents of an alternative stream Zone.Identifier file C:\FirefoxSetup.exe

Remove-Item -Path C:\FirefoxSetup.exe -Stream *- remove all alternative streams associated with the file C:\FirefoxSetup.exe

Remove-Item -Path C:\FirefoxSetup.exe -Stream Zone.Identifier- delete alternative stream Stream Zone.Identifier associated with the file C:\FirefoxSetup.exe.

Security zone information is widely used in group policies, and in particular, by the Windows Attachment Manager, which performs protection functions against malware that may be contained in email attachments or files downloaded from the Internet. The Microsoft website contains a detailed article on how to configure the Attachment Manager and resolve problems associated with it:
Description of the operation of the attachment manager, which is included in the Microsoft Windows system.

In conclusion, I will add that alternative streams are a property of the NTFS file system, and, for example, are not supported in FAT32. Accordingly, when copying files from NTFS to any other file system, alternative streams are discarded.

Alternate Data Stream (AltDS) support was added to NTFS for compatibility with the Macintosh file system HFS, which used a resource stream to store icons and other file information. The use of AltDS is hidden from the user and is not accessible through normal means. Explorer and other applications work with the standard stream and cannot read data from the alternative stream. With AltDS you can easily hide data that cannot be detected by standard system checks. This article will give basic information about the working and definition of AltDS.

Creation of AltDS

Creating an AltDS is very easy. To do this, we will use the command line. First, let's create a base file to which we will attach our streams.

C:\>echo Just a plan text file>sample.txt

C:\>type sample.txt
Just a plan text file

Next, we'll use a colon as an operator to indicate that we'll be using AltDS:

C:\\>echo You can"t see me>sample.txt:secret.txt

You can use the following commands to view the content:

C:\more< sample.txt:secret.txt

or

C:\notepad sample.txt:secret.txt

If everything works well, you will see the text: You can't see me, but when opened from Explorer, this text will not be visible. You can also attach AltDS not only to a file, but also to a folder. To do this, create a folder and attach some kind of some text:

C:\>md stuff
C:\>cd stuff
C:\stuff>echo Hide stuff in stuff>:hide.txt
C:\stuff>dir
Volume in drive C has no label.
Volume Serial Number is 40CC-B506Directory of C:\stuff
09/28/2004 10:19 AM .
09/28/2004 10:19 AM…
0 File(s) 0 bytes2 Dir(s) 12,253,208,576 bytes free
C:\stuff>notepad:hide.txt

Now you know how to view and edit an attached AltDS using Notepad, as well as how to attach it to files and folders.

Hiding and launching applications

Hiding applications using AltDS is as easy as hiding test files. First, let's create the base file again:

Next, let's place our application in a stream, for example I used notepad.exe:

C:\WINDOWS>type notepad.exe>test.txt:note.exe

Now let’s make sure that our file contains the same text:

C:\WINDOWS>type test.txt
Test

And now the fun part, let’s launch our hidden application:

C:\WINDOWS>start .\test.txt:note.exe
C:\WINDOWS>

Since this article is not a complete translation of the article taken, it is formatted as a simple topic. Additional techniques can be found at the link provided.

UPD:

Utilities for working with AltDS (list taken from the article linked above):

LADS - List Alternate Data Streams by Frank Heyne
www.heysoft.de/Frames/f_sw_la_en.htm

Streams.exe from SysInternals.

In this topic, I will look at four types of metadata that can be attached to a file or directory using the file system NTFS. I will describe for what purposes this or that type of metadata can be used, and I will give an example of its use in some Microsoft technology or third-party software.

We will talk about reparse points, object ids and other types of data that a file may contain in addition to its main content.

Object ID this is 64 bytes that can be attached to a file or directory. Of these, the first 16 bytes allow you to uniquely identify a file within the volume and access it not by name, but by identifier. The remaining 48 bytes can contain arbitrary data.

Object IDs have existed in NTFS since Windows 2000. In the system itself, they are used to track the location of the file that a shortcut (.lnk) refers to. Let's say the file referenced by the shortcut has been moved within the volume. When you launch the shortcut, it will still open. A special Windows service, if the file is not found, will attempt to open the file not by its name, but by a previously created and saved identifier. If the file was not deleted and did not leave the volume, it will open, and the shortcut will again point to the file.

Object identifiers were used in the iSwift technology of Kaspersky Anti-Virus version 7. This is how this technology is described: The technology was developed for the NTFS file system. In this system, each object is assigned an NTFS identifier. This identifier is compared with values ​​in a special iSwift database. If the database values ​​with the NTFS identifier do not match, then the object is checked or rechecked if it has been modified.

However, the overabundance of created identifiers caused problems with scanning the disk with the standard chkdsk scanning utility; it took too long. In subsequent versions of Kaspersky Anti-Virus, the use of NTFS Object Id was abandoned.

Reparse Point

In the NTFS file system, a file or directory may contain a reparse point, which is translated into Russian as "reprocess point". Special data is added to the file or directory, the file ceases to be a regular file and can only be processed by a special file system filter driver.

Windows contains reparse point types that can be processed by the system itself. For example, reparse points in Windows implement symbolic links (symlinks) and junction points (junction points), as well as mount points for volumes in a directory.
The reparse buffer attached to a file is a buffer with a maximum size of 16 kilobytes. It is characterized by the presence of a tag that tells the system what type the reparse point belongs to. When using a reparse buffer of your own type, you still need to set the GUID in a special field, and in Microsoft reparse buffers it may not be present.

What types of reprocessing points are there? I will list the technologies that use reparse points. These are Single Instance Storage (SIS) and Cluster Shared Volumes in Windows Storage Server 2008 R2, Hierarchical Storage Management, Distributed File System (DFS), Windows Home Server Drive Extender. These are Microsoft technologies Third-party technologies that use reprocessing points are not mentioned here, although these exist as well.

Extended Attributes

Extended file attributes. It was about them. It is only worth mentioning here that this technology is practically not used under Windows. Of the software I know of, only Cygwin uses extended attributes to store POSIX permissions. A single file on NTFS can have either extended attributes or a reparse point buffer. Simultaneous installation of both is impossible. The maximum size of all extended attributes in one file is 64 KB.

Alternate Data Streams

Additional file streams. Probably everyone already knows about them. I will list the main features of this type of metadata: naming (that is, a file can have several streams, and each has its own name), direct access from the file system (they can be opened using the format “file name, colon, stream name”), unlimited size , the ability to run a process directly from a thread (and the ability to implement it through it).

Used in iStream technology of Kaspersky Anti-Virus. They are used in Windows itself, for example, when downloading a file from the Internet, a Zone.Identifier stream is attached to it, containing information about where the file was received from. After running the executable file, the user may see a message “Unable to verify publisher. Do you really want to run this program?.

This gives the user additional protection against the thoughtless launch of programs obtained from the Internet. This is just one use of streams, and they can store a wide variety of data. The mentioned Kaspersky Anti-Virus stored checksums of each file there, but later this technology was also abandoned for some reason.

Anything else?

There are more security id, plus standard file attributes that are not directly accessible, even though they are also implemented as file streams. And they, and extended attributes, and reparse and object id - all these are file streams from the point of view of the system. There is no point in directly changing the security identifier, shown in the following picture as::$SECURITY_DESCRIPTOR; let the system deal with changing it. The system itself does not provide direct access to other types of streams. So that's it.

Viewing the contents of object id, reparse points, as well as working with extended attributes and alternative file streams is possible using the program

Windows operating systems come with two little-known data hiding features: NTFS data streams (also known as alternate data streams) and Access-based Enumeration (ABE) resource enumeration. Alternate data streams provide the ability to add hidden information to a file, such as file information. You probably won't need to use hidden data streams, but attackers could use this technology against you, so you should be aware of it and how it might work.

As for the ABE method, it can add to your arsenal. This method allows you to make shared resource folders and files invisible to users who do not have permission to access them.

Here's what you need to know about these funds.

Rivers feeding the sea of ​​data

Alternate data streams are a feature of the NTFS file system. They were once included in Windows NT 3.1 to allow NT and Macintosh users to share files.

An NTFS file consists of data streams. This is the standard $DATA data stream, and possibly one or more alternative data streams. Any user with the necessary permissions to the file can see the existing $DATA data stream, can open it, and read and write data to the stream.

An alternate data stream is additional information or files that a user or application can attach to an NTFS file. Only the user who created it knows about the existence of an alternative data stream. Users typically do not know whether an alternate data stream is attached to a file; The point is that neither the content of this stream nor its name are visible. In addition, there is no way to see the change in file size.

There are many ways to use alternative data streams. On Windows, these streams are used to store summary data for documents created by applications that are not included with Microsoft Office, such as plain text (.txt) files. You can enter summary information, such as title, subject, and author information, on the Summary tab of the file's Properties dialog box. This summary data is stored in an alternate data stream, SummaryInformation.

Windows applications such as Encrypting File System (EFS) and Windows Explorer use alternate data streams to append file-specific data to files stored on NTFS-formatted drives. EFS appends encoding and decoding information to encrypted files using alternate data streams, allowing for decentralized encryption and decryption by EFS.

In Windows XP Service Pack 2 (SP2), Microsoft Internet Explorer (IE) uses the Security.Zone alternative data stream to provide security zone classification for files stored on an NTFS volume. As a result, IE has the ability to block user-escalation attacks that can occur in situations where a user downloads malicious code from a non-secure area of ​​the Internet and stores the code on the local hard drive. IE classifies locally stored content into the Local Machine security zone, which provides more rights than the Internet security zone. XP SP2 always checks the Security.Zone data flow before allowing loaded code to take any action on the local system.

Channel for introducing malicious code

What makes alternative data streams noteworthy and dangerous is that their names and contents are not displayed in Windows Explorer. Therefore, the organizers of various types of attacks consider such flows to be a convenient means of hiding data or malicious code that has entered the system. An example of the use of these threads is the VBS.Potok@mm worm. Hackers used an alternate data stream to attach multiple Visual Basic (VB) scripts to an existing ODBC .ini file.

When activated, the worm creates an account with administrative rights and sends itself to addresses that it detects in the Microsoft Outlook address book.

Another danger is that the disk space allocated for alternative data streams is not reflected in Windows Explorer's (file) size and unallocated disk space data. A hacker can use alternate data streams to fill up the file server's disk space, leaving the administrator scratching his head trying to get to the root of the problem. Additionally, it should be noted that the Dir command line utility does not take alternate data streams into account when calculating size data (files and folders). Currently, there is only one Microsoft tool that can take into account alternative data streams when calculating sizes: the Chkdsk utility.

Adding a new thread

Anyone with write access to an NTFS file can use normal operating system commands to append an alternate data stream to the file. For example, the following command creates an alternate data stream mystream, appends mystream to a file named file.txt, and stores the phrase "top secret" in mystream.

echo top secret > file.txt: mystream

You can view the contents of the mystream using the command

As noted above, executable files can be added to alternative data streams. Thus, it is possible to add a hidden copy of the Windows calculator (calc.exe) to the file file.txt. To do this you just need to enter the command

type calc.exe > file.txt: calc.exe

To launch a hidden calculator, enter the command

start .file.txt: calc.exe

You can verify for yourself that alternate data streams and their contents are not displayed in Microsoft tools. Open Windows Explorer and view the properties of file.txt. The actual file size is 112 KB (that's how much space the embedded calc.exe takes up) - but the program will show the file size as 0 KB: there is no information about the embedded file in the $DATA data stream, and Windows Explorer does not have the ability to read information from an alternative data stream .

It is clear that there are many threats associated with alternative data streams, especially in networks where the work of issuing permissions to access NTFS resources is not given due attention and strict access control to Windows servers is not established. There is a simple security mechanism that can prevent hackers from trying to take advantage of alternative data streams - the NTFS access control system. If attackers do not have permission to write data to a file, they will not be able to create alternate data streams and append them to the file.

Detecting Changes

If you feel that hackers have bypassed your permissions, use one of the alternate data stream content discovery tools that have been developed to date. System integrity checkers, such as Tripwire Enterprise and Tripwire for Servers, can detect all changes to the NTFS file system that have occurred on a Windows system, including additions or changes to data stream content.

Sysinternal's Streams program is a free command-line utility that determines the names of alternate data streams attached to files. Figure 1 shows how to use the Streams utility to view the name of the calc.exe data stream that we previously added to the file.txt file. This utility can be downloaded from http://www.sysinternals.com/utilities/streams.html.

Another simple way to detect an alternate data stream is to use Windows Explorer to copy the suspicious file to a non-NTFS drive (say, a FAT drive). Other file systems are not equipped to handle alternative data streams. So if you try to copy an NTFS file with alternate data streams attached to place it on a different file system, NTFS will issue a warning similar to the one shown in Figure 2. But be aware that if you copy this file in the Command Prompt window using the Copy command , Windows will copy it to a non-NTFS file system and delete the data stream without warning.

Hiding shared resources using ABE

ABE is an optional file sharing layer feature that Microsoft first implemented in Windows Server 2003 SP1. It can be used in any Windows shared directory, regardless of the file system on which the shared data is stored. ABE allows administrators to hide folders and files stored on shared resources from users who do not have appropriate permissions to access them at the NTFS level. In other words, we are talking about providing security at the folder level.

In cases where ABE is not used, users connecting to a shared directory will see all files and folders located on the share, including those that they do not have read permissions to read and those that they are blocked from accessing. When a user tries to open a file or folder that they are not allowed to access, the system displays an error message explaining that access is denied. These error messages can be confusing for users, so enabling ABE can help reduce support workload.

However, using ABE also has its disadvantages. Before returning a list of objects in a folder to a client connected to a share, the server must check all ACLs on those objects so that it can determine what data to return. As a result, there may be a significant decrease in system performance, especially when accessing shared resources that contain many objects.

It is advisable to use ABE tools, for example, to configure public resources in user home directories. Instead of creating a hidden share for each user's home directory, you can create one share that contains the home directories of all users under the root home directory folder. Users will connect to this root directory, and you can use ABE as well as NTFS permissions to control the visibility of all users' home directories.

Activating the ABE function

This function uses the new public resource level flag SHI1005_FLAGS_ENFORCE_NAMESPACE_ ACCESS; at the time of writing these lines, it is implemented only in Windows 2003 SP1 and Release 2 (R2). This flag means that you are applying the ABE function to one of the folders.

You can use the Windows Explorer folder properties extensions or the abecmd.exe command line tool to set the flag. Microsoft distributes the ABE Explorer extension and abecmd.exe in the ABE installation package, which is an add-on module for the Windows Server 2003 SP1 platform. The installation package can be downloaded from Microsoft at http://www.microsoft.com/downloads/details.aspx?FamilyId=04A563D 9-78D9-4342-A485-B030AC442084. Because ABE is a server-side extension, it can be used regardless of what version of Windows is installed on the client.

After installing ABE tools on the server, you can set this flag for a particular folder. Right-click the folder, select Properties, go to the Access-based Enumeration tab, and set the Enable access-based enumeration on this shared folder flag, as Figure 3 shows. To apply the ABE function to all shared resources on the system, set the Apply flag this folder"s setting to all existing shared folders on this computer.

The second method is to use the abecmd.exe command line tool. To apply the ABE function to the shareddocs public resource, enter the following command:

abecmd /enable shareddocs

You can use the /all option to enable ABE on all available resources, or use the /disable option to disable ABE.

Access Control

ABE is a simple tool that allows you to limit users' permissions to access only those files that they need to do their work. Users can easily find the files they need because they don't have to wade through unrelated folders, and they don't have to bother support with questions about why files they don't have permission to open won't open.

To protect against alternate data stream hackers, administrators should monitor their public access control settings and use one of the utilities I described to detect hidden alternate data streams as well as changes to the NTFS system.

Jean De Clercq(declercq @hp.com) - employee of the Security Office of Hewlett-Packard. Responsible for identity and security management of Microsoft products. Author of Windows Server 2003 Security Infrastructures (Digital Press).