I know you have a database ...

Nowadays most companies with substantial data have a database system and many big companies also have an ERP system on top of that. These systems are operated by an IT department which supports executives and other business departments by extracting data and generating required reports. The IT department is often too time strained to provide adequate services and is sometimes mired in cultural friction with other personnel. To alleviate this bottleneck issue, many companies use some pre-built user interfaces which allow non-IT users to access data and generate some template reports. But these user interfaces are usually very restrictive in their functions and also requires quite extensive training. Sophisticated business intelligence tools are also created based on a database system or an ERP system for professional analysts to access data and generate custom reports. These tools usually involve steep learning curves and are hard to use for the majority of employees. Another drawback of using these traditional data systems and tools built on top of them is their sub-optimal performance. This is due to the fact that these traditional data systems are designed foremost to store data and ensure data integrity in record-level transactions and not to optimize analysis and report generation which typically involve a large number of records and even the whole dataset.

But do you have a datatop?

A datatop system, which consists of an NETFORCE data engine and an Datatop web user interface, complements a traditional data system by its high speed performance for analysis and report generation, its ease of use for any computer-literate person and its ubiquitous availability through a default web-based user interface. At the core is the NETFORCE data engine which is designed from scratch according to a paradigm which is at the root of any process in analysis and report generation, namely slicing and dicing data followed by aggregation. Slicing and dicing data is achieved through Google-like searches which also recognize wildcarding, word inter-distance, ranges, etc. and can be combined using logical AND/OR/NOT to zero in on any desired subset of data. Special data structures are adopted to optimize searches. Aggregation, which is parallelized to the maximum of CPU and RAM resources available in the server machine or cluster, is dynamic and automatic in response to the user’s choice of granularity levels so that the user can create any desired custom report. Access to the system is through registered accounts with specific privilege levels. Search criteria and report parameters can be saved to rerun later. Reports can be downloaded in MS Excel format. It is also possible to stream data into an Excel template file with pre-built pivot tables and graphics. New features are being added as the system evolves.

Internal applications

• Apart from its basic function of analysis and report generation, a datatop system can also be used to quickly identify data quality issues, such as duplicates, missing values and data errors.
• In connection to the above, applications to detect fraudulent transactions and duplicate payments can be built on top of the system.
• For companies which rely on historical data and demographic data to carry out market targeting campaigns, a process paradigm of slicing and dicing data followed by aggregation at any desired granularity level at a high speed is ideal to improve efficiency and effectiveness. It can be configured to automatically search the parameter space for the most promising market segments to target.

External applications

Supplier Portal

• For historical and technical reasons, big traditional market owners like Walmart and other chain stores have not used their huge sale data to help their suppliers achieve their full potential of marketing and sale. Modern e-commerce platforms like Amazon have been using their big data extensively to help their customers make better purchase decisions. But very little have been done to help their suppliers in terms of information support.
• With appropriate data security and privacy protection in place, a supplier portal can be built where big transaction data centralized at these e-commerce platforms and traditional market owners can be made flow upstream directly to the suppliers as consumer goods flow downstream from them.
• A supplier portal alleviates information asymmetry between suppliers and market owners. It helps suppliers navigate the market and make better decisions. Consumers benefit from lower prices and enhanced market efficiency. It is a win-win-win condition. In 21st century, a supplier portal should be an essential component of the new economy emerging.
• Since each supplier has its own specific interest scope of the market at a particular time and its interest scope changes with time and situations, it is almost impossible for a centralized market owner to provide data and reports specifically tailored to the supplier's specific need at a specific time. The best solution is to have a supplier portal where suppliers can decide on their own what data scope to focus on and what reports to generate, namely a self-served platform. A datatop system is a perfect infrastructure to build such a self-served platform on. This is because, in a datatop system,
  • Slicing and dicing data is achieved through Google-like searches which are easy to use. More sophisticated searches can be built by using Excel-like structures.
  • Reports are generated through aggregation at any desired granularity level at a high speed. This allows a supplier user to quickly iterate to create a desired custom report. Report parameters can be saved to reuse later.
  • A two-tier data structure is in place for each supplier. One is for the global market data which the supplier can access with restricted privileges to protect privacy. The other is the local data consisting of the supplier's sale history without access restriction. Because of this two-tier data structure, each supplier's share of the market can be calculated on the fly at any granularity level. The supplier gains a thorough perspective of its positions in the market so that much better decisions can be made in terms of marketing and sale.