Statistical terms that you don't know yet, but that can leverage your business

A statistical dictionary for those who are not in the area: in today's article, I will translate theoretical terms into practical cases.

In many business correspondence projects, we want to find “correlations” between a specific variable describing an individual and variables. For example, we may know which customers in a historic company after their contracts expire. We can find out what other variables can correlate with a customer in the near future, through a Churn model, for example. Find correlations in the most basic examples of classification tasks and such.
Thus, it is intended to generate 9 more theoretical terms, perfect for real applications, in common situations within the corporate environment, which can help to provide valuable insights. How about we think about these matters?

1. Classification and probability estimation of a class to predict, for each individual in the population, to which (small) set of classes this individual belongs. classes are mutually exclusive. An example of a ranking question would be: “Among all of my company’s customers, who are the respondents to a particular offer?” In this example, both classes can be called: will respond and will not respond. For a classification task, the data mining process of producing a model that, given a new individual, determines the individual's class.

A controlled task is evaluated or performed reliably. The model applied to an individual produced, rather than a class prediction, an attempt to represent the probability (or other quantification of probability) that the individual by class. In our customer response scenario, an assessment model would be able to assess each customer and produce a probability of each response to the offer. Classification and evaluation closely related; a model that can do one can usually be modified to do the other.

2. Value estimation (“Value estimation”) attempts to estimate either for each individual, or predict the numerical value of some variable. An example of demand determined to use the series: “How much service?” The property (to be predicted, here is a service usage variable and a model that can be analyzed and others similar) similar in the population and their usage histories. A specific procedure specifies a model that, given an individual, calculates the value of the variable for that individual. A, is related to classification, however, the two are different. Informally, a classification predicts that the thing will happen, which is something that will happen.


3. Matching by similarity attempts to identify similars based on known data about them. Similarity similarity can be used directly to find similar entities. For example, IBM is interested in finding companies similar to its best business customers, in order to focus its sales force on the best opportunities. They use similarity with “firmographic” databases, which describe the characteristics of companies. Similarity pricing is a basis for one of the most popular methods to make them look like you, in terms of products they liked or bought.) Baseline measures of solutions or other data maintenance tasks, such as units of operation, similarity and grouping.

4. Attempting to group a population by its similarity, but not motivating for any specific purpose. An example of a grouping question would be: “Do our customers form natural groups or segments?” Clustering is useful in the preliminary exploration of domains to exist, as these groups can sometimes suggest other tasks or sometimes their data exploration groups. Clustering is also used as input to decision-making processes focusing on questions such as: what products should we offer or develop? How should our customer service teams (or sales equipment) be structured?

5. Co-occurrence clustering (also known as frequent itemset mining, association rule discovery, and stock portfolio analysis) attempts to find associations between entities based on transactions that involve them. An example of a co-occurrence question would be: What items are commonly purchased together? While clustering analyzes the similarities between objects based on their attributes, co-occurrence clustering considers the similarity of objects based on their joint appearances in transactions. For example, analyzing a supermarket's purchase records may reveal that ground beef is purchased with hot sauce much more often than one might expect. Deciding how to act on this finding might take some creativity, but it might suggest a special promotion, product display, or a combined offer.
Co-occurrence of products in purchases is a common type of grouping known as stock portfolio analysis. Some recommender systems also perform a type of affinity grouping by finding, for example, pairs of books that are often purchased by the same people (“people who bought X also bought Y”). The result of grouping by co-occurrence is a description of the items that occur together. These descriptions usually include statistics on the frequency of the co-occurrence and an estimate of how surprising it is.

6. Profiling (also known as behavior description) attempts to characterize the typical behavior of an individual, group or population. An example of a profiling question would be, “What is typical cell phone usage in this customer segment?” The behavior may not have a simple description; profiling mobile phone usage may require a complex description of overnight and weekend averages, international usage, roaming charges, text content and so on. Behavior can be described generally, for an entire population, or at the level of small groups or even individuals.
Profiling is often used to establish norms of behavior for anomaly detection applications such as fraud detection and monitoring computer system intrusions (such as someone breaking into your iTunes account). For example, if we know what kind of purchases a person normally makes on their credit card, we can determine whether a new charge on the card fits the profile or not. We can use the degree of disparity as a suspect score and raise an alarm if it is too high.

7. Link prediction attempts to predict links between data items, usually suggesting that a link should exist and possibly also estimating the strength of the link. Bond prediction is common in social networking systems: “Since you and João share 10 friends, maybe you would like to be friends with Manuel?” Bond prediction can also estimate the strength of a bond. For example, to recommend movies to customers, you can imagine a graph between customers and the movies they have watched or rated. In the graph, we look for bonds that don't exist between customers and movies, but that we anticipate should and should be strong. These links form the basis of the recommendations.
Current examples of recommender systems: Netflix, Amazon, Spotify,…

8. Data reduction (dimensionality reduction) attempts to take a large set of data and replace it with a smaller set that contains much of the important information from the larger set. It may be easier to handle or process a smaller set of data. What's more, it can reveal the information better. For example, a huge dataset on consumers' movie preferences can be reduced to a much smaller dataset revealing the consumer tastes most evident in the data visualization (eg, viewers' genre preferences). Data reduction usually involves loss of information or even some kind of aggregation. The important thing is the balance for a better understanding.

9. Causal modeling tries to help us understand what events or actions actually influence other people. For example, consider that we use predictive modeling to target ads to consumers, and we observe that target consumers actually buy at a higher rate after they have been targeted. Was this because the ads influenced consumers to buy? Or did the predictive models simply do a good job of identifying consumers who would have purchased anyway?
Causal modeling techniques include those that involve a substantial investment in data, such as randomized controlled experiments (eg, so-called “A/B tests”), as well as sophisticated methods for drawing causal conclusions from observational data. Both experimental and observational methods for causal modeling can often be viewed as “counterfactual” analyses: they try to understand what would be the difference between situations—which are unique to each other—where the “treatment” event (e.g., showing an ad to a particular individual) would and would not happen.

In all cases, a careful data scientist should always include, with a causal conclusion, the exact assumptions that must be made in order for the causal conclusion to hold (these assumptions always exist—always ask). When applying causal modeling, a company needs to weigh the dilemma of increasing investments to reduce the assumptions made versus deciding that the conclusions are good enough given the assumptions. Even in the most careful, randomized, controlled experiment, assumptions are made that could invalidate causal conclusions. The discovery of the “placebo effect” in medicine illustrates a notorious situation where an assumption was ignored in a carefully designed randomized trial.

What did you think? How about investing in intelligence for our business?

Hope to see you in the next article.
Isa