A once do-it-all developer has turned the corner to focus mainly on user experience, visual design, and user interaction. I'm a very technically adept developer, who has decided to hone in on the visual design/interaction side of web applications, and iOS development.
\n", "Quantitative researcher in finance industry, higher degree in mathematics.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Please list the display names of all the users whose accounts were created in the year 2014.", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Please list the x and y values for the second peak in each graph.", "DanTheMan - the cars are all the same year"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I have a list of movies ranked out of 5 based on their income.\\\\n\\\\nI have users ranking the same movies. Not all users rank all movies.\\\\n\\\\nHow can I find out which users have the best correlation between what they ranked and what the income of the movie was? ", "I built a suggestion system, which provides a list of candidate items to users. As users choose some items from the suggested list, the system remembers those choices, and the next time the system reorder the candidate list so the items that were heavily selected before will appear at the top of the list. So the more user use and select items, the better the ranking in the future.\\\\n\\\\nI asked some users to try out the system and collect the time each user spent in selecting items. From my raw calculation, I know that on average, the first half of the users spent 20% more time than the second half of the users. This may suggest that the self-improvement of the system is effective.\\\\n\\\\nBut I don't know which statistical test to use in order to check this hypothesis. Some of my problems include:\\\\n\\\\n - Which distribution to use\\\\n - Since the system is changing as more and more users try it, how to deal with this?\\\\n - I only have 12 users now, it is not that many, how to deal with this?\\\\n"], ["correct the year of the second publication", "improve the list"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I built a suggestion system, which provides a list of candidate items to users. As users choose some items from the suggested list, the system remembers those choices, and the next time the system reorder the candidate list so the items that were heavily selected before will appear at the top of the list. So the more user use and select items, the better the ranking in the future.
\n\nI asked some users to try out the system and collect the time each user spent in selecting items. From my raw calculation, I know that on average, the first half of the users spent 20% more time than the second half of the users. This may suggest that the self-improvement of the system is effective.
\n\nBut I don't know which statistical test to use in order to check this hypothesis. Some of my problems include:
\n\n- \n
- Which distribution to use \n
- Since the system is changing as more and more users try it, how to deal with this? \n
- I only have 12 users now, it is not that many, how to deal with this? \n
The problem was indeed the artificial separation of the year using the 10 factor on latitude and longitudes:
\n\nFirst : this factor was to small for the are studied (the North Sea) so some of the southern and northern points of different years had the same latitudes/longitudes values.\nSecond : this separation is not valid using MEMs because depending of the year Moran's eigenvectors values of the actually same coordinates were positive or negatives, which while selecting MEMs significance created a problem for the models.
\n\nThe solution is to create a grid of coordinates including all the points covered in the time-series and to generate an unique connectivity matrix from which the different Moran's eigenvectors will be computed. The method will not allow to differentiate the different years but is suitable for taking in account spatial arrangement underlying in the data, which create spatial auto-correlation.
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Is it problematic if one predictor in a set accounts for almost all the prediction?", "A regression model whose response variable is the day of year that an annual event (usually) occurs"], ["Currently enrolled in Mindanao State University aiming to graduate the degree of Master in Applied Statistics this year 2014. Hoping to finish this month May, 2014.
\n", "Too Many Interests to list, SE accounts speak for themselves. Christian, Programmer, IT Professional, Embedded enthusiast, chess player, the list goes on.
\n\n\n\n
Interested in all things Embedded?
\n\n![\"Stack](\"http://area51.stackexchange.com/ads/proposal/70800.png\")
If the outcome of interest is number of days since last visit, then first-time users don't provide any information about that outcome, so it seems like leaving out first time visitors is the only thing to do.
\n\nIn any case, coding first time users as 0 reflects a belief that first time users are equivalent to a person who has visited the website before, and whose last visit was 0 days ago, which doesn't make much sense. Therefore, I'd strongly advise against coding in this way, since it would seriously damage interpretability of your results.
I'm monitoring a website where I have two kinds of users (say, A and B). These users are able to contribute to this site in a variety of ways and, for each contribution, its time is logged. What I want is to compare if the number of days that the user is active (contributing in some way) differ between the kinds of users.
\n\nI'm monitoring the website since it's beginning until four months ago. Not all users joined as soon as the site was created nor users that joined at the beginning are still contributing.
\n\nIs survival analysis suitable for this comparison ?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Calculate peak concurrent users on a website", "How many users will connect to my Server?"], ["I'm a Programmer and Web Developer from Tabriz/Iran. I think stackoverflow.com is the best website to find answers to your questions. I have been using this website for a long time but at last I registered and became a member.
\n", "This website is the sole reason I'm still employed
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the display name of the user who has the most number of views?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["@David For the record, Akavall is the one who suggested predicting 1 (since it's the number that occurs most often)", "Any additional info available about when the zeros occurred in the sequence of 20 billion presses, or whether either number is displayed by default (i.e., does the button display something that doesn't appear otherwise, or does it change the display only sometimes), or whether the relationship between the button and the display is causal (i.e., does the button actually affect the display)?"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["The Wikipedia article has accurate information about A/B testing; binary hypothesis testing is another name for A/B testing. A/B testing and split testing are the most widely accepted terms in the business and marketing community. The exact origins of A/B testing are not well known but can be traced back to [Google during the turn of the millennium][1]. \"Google engineers ran their first A/B test at the turn of the millennium to determine the optimum number of results to display on a search engine results page.\"\\\\n\\\\n\\\\n [1]: https://www.pardot.com/blog/abcs-ab-testing/", "In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users).\\\\n\\\\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.\\\\n\\\\nLooking at conversion rate for the segment of users in the 30-35 age group we have:\\\\n\\\\n* P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert?\\\\n* P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion)\\\\n* P(converting) - overall conversion rate (users who converted / all users)\\\\n* P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group\\\\n\\\\nIntuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)].\\\\n\\\\nBut from Bayes theorem:\\\\n\\\\nP(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group)\\\\n\\\\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?"], ["Identify the name of the distribution", "name of the process"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users).
\n\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.
\n\nLooking at conversion rate for the segment of users in the 30-35 age group we have:
\n\n- \n
- P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert? \n
- P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion) \n
- P(converting) - overall conversion rate (users who converted / all users) \n
- P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group \n
Intuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)].
\n\nBut from Bayes theorem:
\n\nP(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group)
\n\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?
\n", "The Wikipedia article has accurate information about A/B testing; binary hypothesis testing is another name for A/B testing. A/B testing and split testing are the most widely accepted terms in the business and marketing community. The exact origins of A/B testing are not well known but can be traced back to Google during the turn of the millennium. \"Google engineers ran their first A/B test at the turn of the millennium to determine the optimum number of results to display on a search engine results page.\"
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Analysis of participant's results to see who is the most different", "What is the name of this normalization?"], ["A once do-it-all developer has turned the corner to focus mainly on user experience, visual design, and user interaction. I'm a very technically adept developer, who has decided to hone in on the visual design/interaction side of web applications, and iOS development.
\n", "You can find my display name on twitter..."], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Among the users who have more than 100 upvotes, how many of them have more then 1 downvotes?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[100, 1], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[100, 1], [1, 32224], [1, 90], ["How many clusters to you want? What do you want the users who are the same cluster to have in common?", "@COOL Yes, that's realistic and it's related to this problem, but it's not the answer. You have accurately computed the chance that a *given* player will collect more than 100 points in 100 throws. The question asks for the chance that the *winner* will collect more than 100 points."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[100, 1], [1, 37], [100, 1], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["Say FB is testing huge, horribly intrusive ads for retirement homes on 90% of its users (call them group A). This angers most people in group A besides 80+ year old users. As a result, most of group A leaves FB permanently. Of course, FB users communicate with each other, so when most of group A leaves, many group B users leave as well. \\\\n\\\\nFB is ultimately left with 80+ year olds who are seeing retirement home ads and some 80+ year olds who do not see them. So, the people in group A who see the ads generate 150% more profit than those in group B. FB might conclude that the ads succeeded, except, the overall profitability of facebook has dropped 90%. \\\\n\\\\n3 questions:\\\\n\\\\nIs this AB test even valid to begin with? \\\\n\\\\nRegardless of your answer to the above, assuming you had to do this test, how would you account for the non-independence of the groups when you interpret the results?\\\\n\\\\nDoes the answer to any of the above change if the test is more subtle, yet still may affect both the experimental and control groups? E.g., FB forces 10% of users to have an inbox that flashes bright colors and induces seizures. So some of those people leave and convince a few of their friends in the control group to leave, but the sample sizes are so small, FB's overall profitability does not decrease significantly as a result of the test.\\\\n", "In many online games, when players complete a difficult task, sometimes a special reward is given which everyone who completed the task can use. this is usually a mount (method of transportation) or another vanity item (items which don't improve the performance of the character and are mainly used for appearance customization).\\\\n\\\\nWhen such a reward is given, the most common way of determining who gets the reward is through random numbers. The game usually has a special command which generates a random (likely pseudorandom, not crypto secure random) number between 1 and 100 (sometimes the player can choose another spread, but 100 is the most common). Each player uses this command, all the players can see who rolled what, and the item is awarded to the person who rolls highest. Most games even have a a built-in system where players just press a button and once everyone pressed their button, the game does the rest automatically.\\\\n\\\\nSometimes, some players generate the same high number and noone beats them. this is usually resolved by those players regenerating their numbers, until there is a unique highest number.\\\\n\\\\nMy question is the following: Assume a random number generator which can generate any number between 1 and 100 with the same probability. Assume that you have a group of 25 players who each generate 1 number with such a random number generator (each with their own seed). You'll have 25 numbers between 1 and 100, with no limitations on how many players roll a specific numbder and no relation between the numbers. What is the chance that the highest generated number is generated by more than 1 player? In other words, what is the likelihood of a tie?"], ["added many more details", "I have corrected the heading to make it more relevant"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 8974], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[100, 1], [1, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [1, 64], [1, 100], ["The way these systems (called recommendation engines) usually work is with item-based filtering. There are various places to read about this. There is a good tutorial in Chapter 2 of Segaran's Programming Collective Intelligence or on Stackexchange (for example this question.) But basically, the problem you have suggested doesn't seem to be a problem because of the way similarities (or affinities) between items are calculated.
\n\nThe basic idea is like this: suppose you want to recommend items to users based on whether other users bought the same item. You have a list of items, call them $A, B, C, \\\\ldots$, which users might buy and a list of users $a, b, c, \\\\ldots$ who each bought some of the items. You make a large table like this
\n\n A B C\n-------\na 1 0 0\nb 1 0 1\nc 0 1 0\nwith a $1$ if user $x$ bought item $Y$ and a $0$ otherwise. For each item, you get a vector, which basically records which users bought it: $A$ would correspond to $(1,1,0)$, $B$ to $(0,0,1)$ and $C$ to $(0,1,0)$ in this example. You then compare items based on how similar their vectors are, perhaps by counting the places where they differ. In this example, suppose someone had bought $A$ and we wanted to tell them what they should buy next. $A$ differs from $B$ in $3$ places and from $C$ in one place, so we should recommend $C$.
\n\nIt is also possible (and better) to use more complicated notions of distance between vectors and to take into account how users might have scored items etc, but this seems to be the general idea.
\n\nComing back to your question, if an item had been bought by everybody, its vector would be $(1,1,1,\\\\ldots)$. It is quite possible that this might not be close to anything and so not get recommended. In the above example, suppose there is a new item $D$ which has been bought by everybody. Suppose a user buys item $C$. The distance of $C$ from $A$ is $1$, the distance of $C$ from $B$ is $2$, and the distance of $C$ from $D$ is $2$, so this user would have $A$ recommended to them instead of the mega-popular $D$.
\n\nIn your scenario, $C$ could be Lars and the Real Girl, $A$ might be Zack and Miri make a Porno, $B$ might be some other fairly niche movie and $D$ might be Lord of the Rings. Someone who had rented only Lars would have Zack recommended to them rather than Lord of the Rings.
\n", "In many online games, when players complete a difficult task, sometimes a special reward is given which everyone who completed the task can use. this is usually a mount (method of transportation) or another vanity item (items which don't improve the performance of the character and are mainly used for appearance customization).
\n\nWhen such a reward is given, the most common way of determining who gets the reward is through random numbers. The game usually has a special command which generates a random (likely pseudorandom, not crypto secure random) number between 1 and 100 (sometimes the player can choose another spread, but 100 is the most common). Each player uses this command, all the players can see who rolled what, and the item is awarded to the person who rolls highest. Most games even have a a built-in system where players just press a button and once everyone pressed their button, the game does the rest automatically.
\n\nSometimes, some players generate the same high number and noone beats them. this is usually resolved by those players regenerating their numbers, until there is a unique highest number.
\n\nMy question is the following: Assume a random number generator which can generate any number between 1 and 100 with the same probability. Assume that you have a group of 25 players who each generate 1 number with such a random number generator (each with their own seed). You'll have 25 numbers between 1 and 100, with no limitations on how many players roll a specific numbder and no relation between the numbers. What is the chance that the highest generated number is generated by more than 1 player? In other words, what is the likelihood of a tie?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Do kernel SVMs have more support vectors than linear ones?", "How can cost decrease by more than 100 percentage points?"], ["Good ideas want as many people to have them as possible
\n", "Ethics are more important than Programming.
\n"], [100, 1], [100, 1], [1, 21], [1, 2792935], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[100, 1], [1, 100], [1, 16], ["2011-02-14", "2010-08-07"], [2050, 2044], [100, 25]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "How many users with more than 10 views created their account after the year 2013?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[10, 30325], [10, 14151], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[10, 67583], [10, 32225], [10, 90], ["[Why can't users with less than 100 reputation answer their own question sooner than 8 hours after asking?](http://meta.stackexchange.com/q/123374/150510)", "How many records do you have in the first 10 months? More importantly, how many fraud records are there in the first 10 months? I will be able to assist with this information! Thanks!"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[10, 118983], [10, 37], [10, 34289], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [10, 16469], ["Say FB is testing huge, horribly intrusive ads for retirement homes on 90% of its users (call them group A). This angers most people in group A besides 80+ year old users. As a result, most of group A leaves FB permanently. Of course, FB users communicate with each other, so when most of group A leaves, many group B users leave as well. \\\\n\\\\nFB is ultimately left with 80+ year olds who are seeing retirement home ads and some 80+ year olds who do not see them. So, the people in group A who see the ads generate 150% more profit than those in group B. FB might conclude that the ads succeeded, except, the overall profitability of facebook has dropped 90%. \\\\n\\\\n3 questions:\\\\n\\\\nIs this AB test even valid to begin with? \\\\n\\\\nRegardless of your answer to the above, assuming you had to do this test, how would you account for the non-independence of the groups when you interpret the results?\\\\n\\\\nDoes the answer to any of the above change if the test is more subtle, yet still may affect both the experimental and control groups? E.g., FB forces 10% of users to have an inbox that flashes bright colors and induces seizures. So some of those people leave and convince a few of their friends in the control group to leave, but the sample sizes are so small, FB's overall profitability does not decrease significantly as a result of the test.\\\\n", "here is my scenario:\\\\n\\\\nPeople record data using sensors, the data is uploaded to a server. Then the users can view their data on a web page. Here is what is in my dataset:\\\\n- I know how often each user uploaded a record to the server\\\\n- I know if and how often that user viewed his records on the website\\\\n\\\\nI want to know how often the users viewed their records on the website.\\\\nNow since not all users have the same life-style and available time to make records and upload them: Would it make sense to standadize the data? If you think yes how would I do that?\\\\n\\\\nWikipedia tells me the standard scores are calculated by z = (x - mean)/standard deviation.\\\\nSo x is my number of record views on the web page, mean is the mean of the views of all users and standard deviation is ... well the standard deviation of all users' views of their data?\\\\n\\\\nAny help is appreciated. If you think I am completely on the wrong path please tell me =)\\\\n"], ["added many more details", "wrong citation year"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [10, 74507], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[10, 34285], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [10, 48], [10, 1257], ["Say FB is testing huge, horribly intrusive ads for retirement homes on 90% of its users (call them group A). This angers most people in group A besides 80+ year old users. As a result, most of group A leaves FB permanently. Of course, FB users communicate with each other, so when most of group A leaves, many group B users leave as well.
\n\nFB is ultimately left with 80+ year olds who are seeing retirement home ads and some 80+ year olds who do not see them. So, the people in group A who see the ads generate 150% more profit than those in group B. FB might conclude that the ads succeeded, except, the overall profitability of facebook has dropped 90%.
\n\n3 questions:
\n\nIs this AB test even valid to begin with?
\n\nRegardless of your answer to the above, assuming you had to do this test, how would you account for the non-independence of the groups when you interpret the results?
\n\nDoes the answer to any of the above change if the test is more subtle, yet still may affect both the experimental and control groups? E.g., FB forces 10% of users to have an inbox that flashes bright colors and induces seizures. So some of those people leave and convince a few of their friends in the control group to leave, but the sample sizes are so small, FB's overall profitability does not decrease significantly as a result of the test.
\n", "I have the following data:
\n\n1000 users were invited\n 990 emails delivered successfully\n 200 visited the system\n 60 tried to create an account \n 58 tried to create an account with a valid email\n 50 successfully created an account (i.e. chose a secure password)\n 40 participated by voting\n 10 posted something\nWhile I could use a bar graph, the proportions of 60:58:50 would likely be lost. If I'd use a logarithmic scale, the proportions would be distorted and the feeling of numbers would be lost.
\n\nIs there a good way to represent both the (significant) loss in user attention due to the number of hurdles as well as the effect of each hurdle?
\n"], [10, 16481], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["CCA/KCCA for more than two views", "Calculating 10 year probability using hazard ratios?"], ["20 years in programming
\n
\nProgramming with .NET since 2002, after more than a decade of C++ programming.
\n
\nMVP Visual C# since 2003
\n
\nLead developer of the tool NDepend
I'm currently working half-time as a Research Fellow in the Statistics Department at the University of Warwick and half-time as a freelance statistical/R consultant.
\n\nI spend most of my time working in R, but also use markup languages lots quite a bit.
I've finally managed to log back in to my Stackoverflow account after a long time out, so hopefully the account will show more activity soon!
\n"], [10, 117], [10, 107], [10, 20], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[10, 14860], [10, 3330], [10, 16], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "How many posts does the user csgillespie own?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["@TanyaKiddle How many levels does your factor have? How did you create the contrast matrix?", "How many different terms does the union of your sets contain?"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I have logs for users and posts in a blog platform for 3 years. I can easily find out how many posts each user have made per day/month/year/etc.\\\\n\\\\nWhat I want to find out is if the frequency of posts is growing or not, in other words, if the users are contributing more over time or not. New users could join at any time and old ones could abandon the system (only the first one is logged). Is a trend estimation (http://en.wikipedia.org/wiki/Trend_estimation) suitable for this ? \\\\n\\\\nMore over, is there a python package for this kind of time analyse ? \\\\n", "Let's say I have 300 restaurants that I want to compare to each other on the basis of a \"similarity score\". To try and determine similarity scores, I pick a reference restaurant and pick 3 other restaurants at random and ask a user \"Which of these 3 restaurants is most similar to the reference restaurant?\". The user selects one of the 3 restaurants and the data is recorded. How many times does this question have to be answered before I can reasonably start to use the data to determine similarity between restaurants?"], ["How to enter data.", "many spelling errors"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I have logs for users and posts in a blog platform for 3 years. I can easily find out how many posts each user have made per day/month/year/etc.
\n\nWhat I want to find out is if the frequency of posts is growing or not, in other words, if the users are contributing more over time or not. New users could join at any time and old ones could abandon the system (only the first one is logged). Is a trend estimation (http://en.wikipedia.org/wiki/Trend_estimation) suitable for this ?
\n\nMore over, is there a python package for this kind of time analyse ?
\n", "Let's say I have 300 restaurants that I want to compare to each other on the basis of a \"similarity score\". To try and determine similarity scores, I pick a reference restaurant and pick 3 other restaurants at random and ask a user \"Which of these 3 restaurants is most similar to the reference restaurant?\". The user selects one of the 3 restaurants and the data is recorded. How many times does this question have to be answered before I can reasonably start to use the data to determine similarity between restaurants?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["How many parameters does a HM-SVM require?", "How many user interactions do I need to perform search relevance experiment?"], ["Hi.
\n\n\n\n
SO Milestones:
\n\n- \n
- 2nd user to get the silver stl badge. \n
- 3rd user to get the silver templates badge. \n
- 15th user to get the gold c++ badge. \n
- 27th user to get the silver c badge. \n
- 102nd user to get the legendary badge. \n
\n\n
Favorite answers:
\n\n- \n
- What is the copy-and-swap idiom? \n
- How far to go with a strongly typed language? \n
- [C++] Why doesn't delete destroy anything ? \n
- Advantage of using forward \n
- When does invoking a member function on a null instance result in undefined behavior? \n
- Is it possible to hash pointers in portable C++03 code? (Value vs. value representation) \n
- String literals not allowed as non type template parameters \n
- C++: How do I prevent a function from accepting a pointer that is allocated in-line? \n
- Why do I see strange values when I print uninitialized variables? \n
- Question about pointer increment \n
- Building boost::options from a string/boost::any map (Type-erasure) \n
Developer who does not code and an Architect who does not day-dream !!
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Please list the titles of the posts owned by the user csgillespie?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Sorry, I've never user the function. Please consult the documentation", "Please list the x and y values for the second peak in each graph."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["**Online resources:**\\\\n\\\\n[UCLA](http://www.ats.ucla.edu/stat/sas/) has compiled a list of resources \\\\n\\\\nYou can also use Google search term such as `SAS learning filetype:pdf` to look for tutorials. For instance, I found this [great list]( http://analytics.ncsu.edu/sesug/2003/TD02-Stojanovic.pdf) made by Stojanovic.\\\\n\\\\nSAS User Group International (SUGI) also posts [conference proceedings online](http://support.sas.com/events/sasglobalforum/previous/online.html). Each paper addresses a particular topic in various degree of complexity. Most of them are structured as a mini tutorial, which is user friendly. (Tip: the site requires a search term, if you're not sure, just leave it empty and click the search button.)\\\\n\\\\nSAS also put a lot of manuals online. For instance, if you Google search `proc reg filetype:pdf`, you'll find the manual hosted at support.sas.com. You can get a good understanding on handy procedures such as `proc sql` and `ods` (Output Delivery System) this way.\\\\n\\\\n**Book:**\\\\n\\\\nEven online resources are everywhere, there are a couple SAS books that I still read all the time. First, if money allows, I'd suggest get a few titles by [Cody](http://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=Ron+Cody). Among which, I'd suggest strongly [Learning SAS by Example](http://www.amazon.com/Learning-SAS-Example-Programmers-Guide/dp/1599941651) (Here is an [excerpt](http://www.sas.com/storefront/aux/en/splearnexample/60864_excerpt.pdf)), [Applied Statistics and the SAS Programming Language](http://www.amazon.com/Applied-Statistics-Programming-Language-Edition/dp/0131465325), and perhaps [Cody's Data Cleaning Techniques Using SAS](http://www.amazon.com/Codys-Cleaning-Techniques-Second-Edition/dp/1599946599), but this last one is not essential at the early stage.\\\\n\\\\nAnother classic you should check out is the [The Little SAS Book](http://www.amazon.com/The-Little-SAS-Book-Edition/dp/1612903436). It covers a lot of basics, and I'd recommend paring this with Learning SAS by Example; that way you should be able to tackle most of the basic stuff.", "**Online resources:**\\\\n\\\\n[UCLA](http://www.ats.ucla.edu/stat/sas/) has compiled a list of resources \\\\n\\\\nYou can also use Google search term such as `SAS learning filetype:pdf` to look for tutorials. For instance, I found this [great list]( http://analytics.ncsu.edu/sesug/2003/TD02-Stojanovic.pdf) made by Stojanovic.\\\\n\\\\nSAS User Group International (SUGI) also posts [conference proceedings online](http://support.sas.com/events/sasglobalforum/previous/online.html). Each paper addresses a particular topic in various degree of complexity. Most of them are structured as a mini tutorial, which is user friendly. (Tip: the site requires a search term, if you're not sure, just leave it empty and click the search button.)\\\\n\\\\nSAS also put a lot of manuals online. For instance, if you Google search `proc reg filetype:pdf`, you'll find the manual hosted at support.sas.com. You can get a good understanding on handy procedures such as `proc sql` and `ods` (Ooutput Delivery System) this way.\\\\n\\\\n**Book:**\\\\n\\\\nEven online resources are everywhere, there are a couple SAS books that I still read all the time. First, if money allows, I'd suggest get a few titles by [Cody](http://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=Ron+Cody). Among which, I'd suggest strongly [Learning SAS by Example](http://www.amazon.com/Learning-SAS-Example-Programmers-Guide/dp/1599941651) (Here is an [excerpt](http://www.sas.com/storefront/aux/en/splearnexample/60864_excerpt.pdf)), [Applied Statistics and the SAS Programming Language](http://www.amazon.com/Applied-Statistics-Programming-Language-Edition/dp/0131465325), and perhaps [Cody's Data Cleaning Techniques Using SAS](http://www.amazon.com/Codys-Cleaning-Techniques-Second-Edition/dp/1599946599), but this last one is not essential at the early stage.\\\\n\\\\nAnother classic you should check out is the [The Little SAS Book](http://www.amazon.com/The-Little-SAS-Book-Edition/dp/1612903436). It covers a lot of basics, and I'd recommend paring this with Learning SAS by Example; that way you should be able to tackle most of the basic stuff."], ["improve the list", "formatted the numbered list"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["Online resources:
\n\nUCLA has compiled a list of resources
\n\nYou can also use Google search term such as SAS learning filetype:pdf to look for tutorials. For instance, I found this great list made by Stojanovic.
SAS User Group International (SUGI) also posts conference proceedings online. Each paper addresses a particular topic in various degree of complexity. Most of them are structured as a mini tutorial, which is user friendly. (Tip: the site requires a search term, if you're not sure, just leave it empty and click the search button.)
\n\nSAS also put a lot of manuals online. For instance, if you Google search proc reg filetype:pdf, you'll find the manual hosted at support.sas.com. You can get a good understanding on handy procedures such as proc sql and ods (Output Delivery System) this way.
Book:
\n\nEven online resources are everywhere, there are a couple SAS books that I still read all the time. First, if money allows, I'd suggest get a few titles by Cody. Among which, I'd suggest strongly Learning SAS by Example (Here is an excerpt), Applied Statistics and the SAS Programming Language, and perhaps Cody's Data Cleaning Techniques Using SAS, but this last one is not essential at the early stage.
\n\nAnother classic you should check out is the The Little SAS Book. It covers a lot of basics, and I'd recommend paring this with Learning SAS by Example; that way you should be able to tackle most of the basic stuff.
\n", "I suggest to use Expected Utility or R-score. Assume your model has created an ordered list of recommendations where the first item is the one the user is most likely and the last is the one the user is least likely interested in. Let's say that this recommendations are specified by $r_i$ where $i$ is the position in the list.
\n\nExpected utility for a particular user u is now defined as:\n$R_u=\\\\sum_{i=1}^{n}\\\\frac{f_u(r_i)}{2^{\\\\frac{i-1}{\\\\alpha-1}}}$
\n\nwhere ...
\n\n- \n
- $f_u(r_i)=1$, if the recommended item $r_i$ is the user's library, else 0 \n
- $\\\\alpha$ specifies the slope of the decay. The smaller alpha, the greater the slope. \n
This metric measures how likely it is that a user will view a recommended item, assuming a list of items is recommended to him. If the item of interest is placed at a very high position, it is unlikely that the user will bother to scroll/look that far, even if it exactly what he/she wanted. In this sense, $\\\\alpha$ specifies how patient the average user is.
\n\nTo calculate the R-score for a set of users, it is recommended to normalize the R-score per user beforehand. The resulting score is
\n\n$R =\\\\sum_u\\\\frac{R_u}{R_u^*}$
\n\nwhere $R_u^*$ is the maximum R-score you can achieve for one user. That is, given a user has k items in his library you want to predict, the items at the first k positions are exactly these ones.
\n\nFor more metrics or as a general read I recommend Evaluating Recommendations Systems by Shani & Gunarwadana
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What is the estimation bias of the top estimate in a list sorted by value?", "How to do the prediction of user action?"], ["Video gamer, avid reader, feminist, owned by cats, migraines.
\n\nSome game IDs:
\n\n- \n
- Warcraft/Hearthstone BT Zaralynda#1825 \n
- 3DS 5257-9871-9599 (AC:NL, P:XY) \n
Ubuntu user since the end of 2008.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Who is the owner of the post \"Eliciting priors from experts\"?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Sorry, I edited the post so that it has the priors that I meant.", "@Komal I tried, but the \"question owner is blocked from asking questions\" on SO. Sorry."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I can't post a comment (the appropriate place for this comment) as I don't have enough reputation, but the answer accepted as the best answer by the question owner misses the point.\\\\n\\\\n\"If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss.\"\\\\n\\\\nThe likelihood is a loss function. Maximising likelihood is the same as minimising a loss function: the deviance, which is just -2 times the log-likelihood function. Similarly finding a least squares solution is about minimising the loss function describing the residual sum of squares. \\\\n\\\\nBoth ML and stats use algorithms to optimise the fit of some function (in the broadest terms) to data. Optimisation necessarily involves minimising some loss function.", "No, a Gamma(0,0) is not equivalent to the Jeffreys prior of the Poisson and Exponential rates. The Gammao(0,0) (actually a $Gamma(\\\\epsilon,\\\\epsilon)$ with $\\\\epsilon\\\\approx 0$) is typically used since it \"resembles\" the shape of the Jeffreys prior. Also, it became popular since it is implemented in WINBUGS.\\\\n\\\\nSee also: http://stats.stackexchange.com/questions/50016/exponential-distribution-rate-bayesian-prior\\\\n\\\\nAlthough the Gamma prior is decreasing, the tails of this and the Jeffreys priors are different. Moreover, the Jeffreys prior of the Exponential is $1/\\\\lambda$, while in the [Poisson case](http://en.wikipedia.org/wiki/Jeffreys_prior) is $\\\\lambda^{-1/2}$, then, the claimed resemblance is not theoretically justified. In practice, however, with large or moderate samples, the influence of the prior is usually negligible. For this simple models, you can simply use the exact expression of the Jeffreys priors since they produce proper posteriors and they are easy to sample from using basic MCMC algorithms.\\\\n"], ["Formating the post", "clarified the main issue of the post in the title"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I can't post a comment (the appropriate place for this comment) as I don't have enough reputation, but the answer accepted as the best answer by the question owner misses the point.
\n\n\"If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss.\"
\n\nThe likelihood is a loss function. Maximising likelihood is the same as minimising a loss function: the deviance, which is just -2 times the log-likelihood function. Similarly finding a least squares solution is about minimising the loss function describing the residual sum of squares.
\n\nBoth ML and stats use algorithms to optimise the fit of some function (in the broadest terms) to data. Optimisation necessarily involves minimising some loss function.
\n", "As mentioned by the OP, this is probably not the right place for expert advice on legal issues, but we all have to live with such things as software licenses and try not to get into trouble, so here are a few things that I have learned.
\n\n- \n
- On the NR homepage you can find the license information and information on redistribution. This is solely a copyright issue regarding the source code provided in the books. The algorithms themselves are not copyrighted, and to my understanding you can't get into trouble by implementing and sharing an algorithm that happens to be in NR unless your implementation is derived from the source code distributed with NR. \n
- Hmm, nothing is free ... There is almost always a copyright holder. If the paper is published, the copyright may be transferred to the journal, but in some cases the author retain the right to distribute the paper via his or her homepage, say. Patent law is a completely different ball game, but I actually don't know of any examples related to statistical and machine learning algorithms where a patent protection of the algorithm was a problem. Hence, to me, the most important aspect regarding algorithms in papers, whether published in journals or on the authors homepage, is not to violate the copyright. \n
- A trademark is a third thing. To my understanding it protects only the name or symbol. So you just can't implement a program and call it \"Random Forests\", but you can implement and distribute the random forest algorithm. \n
- I have no idea if Frank Masci broke the law, but it is a copyright issue as I see it. Who holds the copyright and which rights did the copyright holder give to others? \n
From my point of view, though violations of patent rights might be serious, the more important issue for the average statistician is that of copyright. If you implement an algorithm from a paper from scratch and cite the paper I don't see any obvious problem with distributing the implementation regardless of the media. But you might think about what is the best way for you to use your copyright on the implementation. I don't know anything about how blogs are generally copyrighted, but if the copyright is transferred to the blog owner automatically, it might, in principle, be a bad idea to post hours of valuable implementations on a blog and thereby effectively give up the copyright of the work to somebody else.
\n\nIf you, on the other hand, modify an existing implementation, you could run into problems with the copyright license. If the license for the original implementation is GPL, the license for the redistribution has to be GPL. Hence, you have to distribute in a way so that the distribution can be under GPL. This works the other way too. If you want to distribute an implementation under GPL but have to rely on a library that is not distributed under the GNU license, then you might not be able to include the library in your distribution $-$ even if the library is open source and \"free\". The library might be distributed under a copyright license that is incompatible with GPL.
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Eliciting priors from experts", "Who is the best writer among statisticians?"], ["What do I look for in a post? Clarity and Contribution.
\n\nNO vote: If the post is clear, but makes no contribution OR if the post is unclear, but makes a contribution (though I will try to improve the clarity either by comments or direct edit).
\n\nUP vote: If the post is clear and makes a contribution.
\n\nDOWN vote: If the post is unclear and makes no contribution (though I will try to see if I've missed something by commenting).
\n", "I'm a post doc in the area of computational neuroscience.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the title of the post that is owned by csgillespie and has the highest popularity?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Very clear answer. I don't feel very comfortable in predicting categories by picking up the category with the highest probability. What is your opinion on picking the category by taking the weighted average of the probabilities and the corresponding classes?", "What is the \"conclusion\" to which the title refers?"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["Your title and some of the body of your post aren't consistent.\\\\n\\\\nYour title (and last sentence for example) is about skewness. I take that to be the central issue.\\\\n\\\\nHowever, one section in the body of your post asks about symmetry and makes the (incorrect) assertion that: \\\\n> Symmetric distributions have skewness = 0\\\\n\\\\nThis is not true in general! It's quite possible to have symmetric distributions that *don't* have zero third moment.\\\\n\\\\nYou ask about the moment-skewness coefficient of a point mass (in the title and in the later part of the question). [If your real question is \"are point masses symmetric\", then 'yes' is a perfectly sensible, (almost) trivial answer, from several approaches. But I assume the title Q is the main Q.]\\\\n\\\\nNote that $0/0$ is undefined. Let me explain why \"undefined\" is the right answer for the skewness in this case.\\\\n\\\\nWhen you have a ratio where the numerator and denominator are both 0, it's sometimes possible to consider the limit of the ratio (that is, to set up a sequence $a_i/b_i$ where $a_i$ and $b_i$ both go to $0$, and see what happens to the ratio as $i\\\\to \\\\infty$), and so perhaps (if the $a$'s and $b$s must be connected in a way that gives it one) obtain a value for the limit of the ratio that would be a reasonable answer to the title question.\\\\n\\\\nBut no such limit exists in this case.\\\\n\\\\nLet's consider a sub-family of shifted-Gamma distributions with fixed shape parameter $\\\\alpha$ and varying scale parameter, $\\\\beta$, with the shift, $\\\\theta$, chosen so that $\\\\theta=c-\\\\alpha\\\\beta$ for some constant $c$. \\\\n\\\\nPlainly the expectation of each distribution is $c$.\\\\n \\\\nThe standardized-third-moment skewness coefficient of a gamma distribution (and hence of a shifted gamma) is $\\\\frac{2}{\\\\sqrt\\\\alpha}$. \\\\n\\\\nSince we hold $\\\\alpha$ fixed, the skewness of every member of the family is the same. \\\\n\\\\nNow consider what happens as the scale parameter, $\\\\beta\\\\to 0$. In the limit, the distribution is a point mass at $c$, but at every point along the way, the skewness is $\\\\frac{2}{\\\\sqrt\\\\alpha}$.\\\\n\\\\n$\\\\hspace{0.7 cm}$ ![enter image description here][1]\\\\n$^{\\\\text{ The first few members of a sequence of distribution functions where expectation }=4,\\\\text{ variance }\\\\to 0 \\\\text{ but } \\\\gamma=\\\\sqrt{2}}$\\\\n\\\\nHence, in the limit, the skewness for the point-mass distribution is also $\\\\frac{2}{\\\\sqrt\\\\alpha}$.\\\\n\\\\nBut $\\\\alpha$ could have been chosen to be *any value at all*, $>0$. Hence, every positive value for skewness is potentially available as a limit for the skewness at the point-mass.\\\\n\\\\nNow consider that we could also flip this family about $c$ and have another family with negative skewness.\\\\n\\\\nHence, every negative value for skewness is potentially available as a limit for the skewness at the point-mass.\\\\n\\\\nBy similar argument, I can construct a symmetric family whose third moment is 0 (say a Gaussian family) with the same point mass limit.\\\\n\\\\nHence, $0$ is available as a limit for the skewness at the point-mass.\\\\n\\\\nSimilarly I can also obtain $\\\\infty$, $-\\\\infty$ and \"undefined\" (by taking a family with the relevant skewness coefficient and taking the limit in similar fashion).\\\\n\\\\nAs a result, every conceivable value (including no value at all) of the skewness coefficient is consistent with a point mass, simply by taking the limit of a different sequence of distributions.\\\\n\\\\nSo the *right* answer to the title question is \"it's undefined\", just as $0/0$ would have suggested to us.\\\\n\\\\n[An argument from symmetry of the cdf of the point mass is *irrelevant* to the title question, which wasn't about symmetry, but about third moment skewness. If you think symmetry speaks to that question, note that the Cauchy distribution is symmetric, but its skewness isn't 0.]\\\\n\\\\n\\\\n [1]: http://i.stack.imgur.com/YcXSn.png", "Your title and some of the body of your post aren't consistent.\\\\n\\\\nYour title (and last sentence for example) is about skewness. I take that to be the central issue.\\\\n\\\\nHowever, one section in the body of your post asks about symmetry and makes the (incorrect) assertion that: \\\\n> Symmetric distributions have skewness = 0\\\\n\\\\nThis is not true in general! It's quite possible to have symmetric distributions that *don't* have zero third moment. (There are also asymmetric distributions with zero third moment. Moment-skewness and symmetry aren't as closely related as we might wish them to be.)\\\\n\\\\nYou ask about the moment-skewness coefficient of a point mass (in the title and in the later part of the question). [If your real question is \"are point masses symmetric\", then 'yes' is a perfectly sensible, (almost) trivial answer, from several approaches. But I assume the title Q is the main Q.]\\\\n\\\\nNote that $0/0$ is undefined. Let me explain why \"undefined\" is the right answer for the skewness in this case.\\\\n\\\\nWhen you have a ratio where the numerator and denominator are both 0, it's sometimes possible to consider the limit of the ratio (that is, to set up a sequence $a_i/b_i$ where $a_i$ and $b_i$ both go to $0$, and see what happens to the ratio as $i\\\\to \\\\infty$), and so perhaps (if the $a$'s and $b$s must be connected in a way that gives it one) obtain a value for the limit of the ratio that would be a reasonable answer to the title question.\\\\n\\\\nBut no such limit exists in this case.\\\\n\\\\nLet's consider a sub-family of shifted-Gamma distributions with fixed shape parameter $\\\\alpha$ and varying scale parameter, $\\\\beta$, with the shift, $\\\\theta$, chosen so that $\\\\theta=c-\\\\alpha\\\\beta$ for some constant $c$. \\\\n\\\\nPlainly the expectation of each distribution is $c$.\\\\n \\\\nThe standardized-third-moment skewness coefficient of a gamma distribution (and hence of a shifted gamma) is $\\\\frac{2}{\\\\sqrt\\\\alpha}$. \\\\n\\\\nSince we hold $\\\\alpha$ fixed, the skewness of every member of the family is the same. \\\\n\\\\nNow consider what happens as the scale parameter, $\\\\beta\\\\to 0$. In the limit, the distribution is a point mass at $c$, but at every point along the way, the skewness is $\\\\frac{2}{\\\\sqrt\\\\alpha}$.\\\\n\\\\n$\\\\hspace{0.7 cm}$ ![enter image description here][1]\\\\n$^{\\\\text{ The first few members of a sequence of distribution functions where expectation }=4,\\\\text{ variance }\\\\to 0 \\\\text{ but } \\\\gamma=\\\\sqrt{2}}$\\\\n\\\\nHence, in the limit, the skewness for the point-mass distribution is also $\\\\frac{2}{\\\\sqrt\\\\alpha}$.\\\\n\\\\nBut $\\\\alpha$ could have been chosen to be *any value at all*, $>0$. Hence, every positive value for skewness is potentially available as a limit for the skewness at the point-mass.\\\\n\\\\nNow consider that we could also flip this family about $c$ and have another family with negative skewness.\\\\n\\\\nHence, every negative value for skewness is potentially available as a limit for the skewness at the point-mass.\\\\n\\\\nBy similar argument, I can construct a symmetric family whose third moment is 0 (say a Gaussian family) with the same point mass limit.\\\\n\\\\nHence, $0$ is available as a limit for the skewness at the point-mass.\\\\n\\\\nSimilarly I can also obtain $\\\\infty$, $-\\\\infty$ and \"undefined\" (by taking a family with the relevant skewness coefficient and taking the limit in similar fashion).\\\\n\\\\nAs a result, every conceivable value (including no value at all) of the skewness coefficient is consistent with a point mass, simply by taking the limit of a different sequence of distributions.\\\\n\\\\nSo the *right* answer to the title question is \"it's undefined\", just as $0/0$ would have suggested to us.\\\\n\\\\n[An argument from symmetry of the cdf of the point mass is *irrelevant* to the title question, which wasn't about symmetry, but about third moment skewness. If you think symmetry speaks to that question, note that the Cauchy distribution is symmetric, but its skewness isn't 0.]\\\\n\\\\n\\\\n [1]: http://i.stack.imgur.com/YcXSn.png"], ["Orthant is the correct term that suits the content of the question", "corrected the title so it is consistent with the quote that motivated the question"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["Your title and some of the body of your post aren't consistent.
\n\nYour title (and last sentence for example) is about skewness. I take that to be the central issue.
\n\nHowever, one section in the body of your post asks about symmetry and makes the (incorrect) assertion that:
\n\n\n\n\nSymmetric distributions have skewness = 0
\n
This is not true in general! It's quite possible to have symmetric distributions that don't have zero third moment. (There are also asymmetric distributions with zero third moment. Moment-skewness and symmetry aren't as closely related as we might wish them to be.)
\n\nYou ask about the moment-skewness coefficient of a point mass (in the title and in the later part of the question). [If your real question is \"are point masses symmetric\", then 'yes' is a perfectly sensible, (almost) trivial answer, from several approaches. But I assume the title Q is the main Q.]
\n\nNote that $0/0$ is undefined. Let me explain why \"undefined\" is the right answer for the skewness in this case.
\n\nWhen you have a ratio where the numerator and denominator are both 0, it's sometimes possible to consider the limit of the ratio (that is, to set up a sequence $a_i/b_i$ where $a_i$ and $b_i$ both go to $0$, and see what happens to the ratio as $i\\\\to \\\\infty$), and so perhaps (if the $a$'s and $b$s must be connected in a way that gives it one) obtain a value for the limit of the ratio that would be a reasonable answer to the title question.
\n\nBut no such limit exists in this case.
\n\nLet's consider a sub-family of shifted-Gamma distributions with fixed shape parameter $\\\\alpha$ and varying scale parameter, $\\\\beta$, with the shift, $\\\\theta$, chosen so that $\\\\theta=c-\\\\alpha\\\\beta$ for some constant $c$.
\n\nPlainly the expectation of each distribution is $c$.
\n\nThe standardized-third-moment skewness coefficient of a gamma distribution (and hence of a shifted gamma) is $\\\\frac{2}{\\\\sqrt\\\\alpha}$.
\n\nSince we hold $\\\\alpha$ fixed, the skewness of every member of the family is the same.
\n\nNow consider what happens as the scale parameter, $\\\\beta\\\\to 0$. In the limit, the distribution is a point mass at $c$, but at every point along the way, the skewness is $\\\\frac{2}{\\\\sqrt\\\\alpha}$.
\n\n$\\\\hspace{0.7 cm}$ ![\"enter](\"http://i.stack.imgur.com/YcXSn.png\")
\n$^{\\\\text{ The first few members of a sequence of distribution functions where expectation }=4,\\\\text{ variance }\\\\to 0 \\\\text{ but } \\\\gamma=\\\\sqrt{2}}$
Hence, in the limit, the skewness for the point-mass distribution is also $\\\\frac{2}{\\\\sqrt\\\\alpha}$.
\n\nBut $\\\\alpha$ could have been chosen to be any value at all, $>0$. Hence, every positive value for skewness is potentially available as a limit for the skewness at the point-mass.
\n\nNow consider that we could also flip this family about $c$ and have another family with negative skewness.
\n\nHence, every negative value for skewness is potentially available as a limit for the skewness at the point-mass.
\n\nBy similar argument, I can construct a symmetric family whose third moment is 0 (say a Gaussian family) with the same point mass limit.
\n\nHence, $0$ is available as a limit for the skewness at the point-mass.
\n\nSimilarly I can also obtain $\\\\infty$, $-\\\\infty$ and \"undefined\" (by taking a family with the relevant skewness coefficient and taking the limit in similar fashion).
\n\nAs a result, every conceivable value (including no value at all) of the skewness coefficient is consistent with a point mass, simply by taking the limit of a different sequence of distributions.
\n\nSo the right answer to the title question is \"it's undefined\", just as $0/0$ would have suggested to us.
\n\n[An argument from symmetry of the cdf of the point mass is irrelevant to the title question, which wasn't about symmetry, but about third moment skewness. If you think symmetry speaks to that question, note that the Cauchy distribution is symmetric, but its skewness isn't 0.]
\n", "I am preparing to analyze some data and I have a question about how to go about it. It is a 39 question survey with 5-item Likert scale responses. I will be using the responses to separate the participants into the category that they score highest in for further analysis. The problem is that Category A corresponds to 14 questions, Category B has 11 questions, and Category C has 14 questions. What is the best way to go about determining which one the participant scores highest in, since the # of questions is not equal? Will it work to take their total possible points in each category and divide it by the number of points possible, then compare their percentages across the three categories to see which is the highest?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What is the best way to detect trends in item popularity?", "What is the probability that this person is female?"], ["What do I look for in a post? Clarity and Contribution.
\n\nNO vote: If the post is clear, but makes no contribution OR if the post is unclear, but makes a contribution (though I will try to improve the clarity either by comments or direct edit).
\n\nUP vote: If the post is clear and makes a contribution.
\n\nDOWN vote: If the post is unclear and makes no contribution (though I will try to see if I've missed something by commenting).
\n", "Another PHP programmer responsible for things like the MicroMVC PHP framework.\\\\\\\\n\\\\\\\\nYou may notice that some of my questions run very deep into the heart of certain issues because I strive for the highest optimization that is possible in my apps."], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the display name of the user who is the owner of the most valuable post?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["@Shashank aaaah, now we are getting somewhere. So it is basically a content based recommender with the owner / user of the device being the only one which provides explicit feedback / rating ?", "Hi Patrick, you don't need to sign your name at the end of your posts. You can change your user name by clicking it at the top of the page, then pressing 'edit'."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users).\\\\n\\\\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.\\\\n\\\\nLooking at conversion rate for the segment of users in the 30-35 age group we have:\\\\n\\\\n* P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert?\\\\n* P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion)\\\\n* P(converting) - overall conversion rate (users who converted / all users)\\\\n* P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group\\\\n\\\\nIntuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)].\\\\n\\\\nBut from Bayes theorem:\\\\n\\\\nP(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group)\\\\n\\\\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?", "In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users ).\\\\n\\\\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.\\\\n\\\\nLooking at conversion rate for the segment of users in the 30-35 age group we have:\\\\n\\\\nP( conversion | in the 30-35 age group ) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert?\\\\nP( in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion)\\\\nP( converting ) - overall conversion rate (users who converted / all users)\\\\nP(in the 30-35 age group) - proportion of all users who are n the 30-35 age group\\\\n\\\\nIntuitively we might think that a new user in the 30-35 age group's probability to convert ( P( conversion | in the 30-35 age group ) ) is the same as the conversion rate (P( in the 30-35 age group | conversion)).\\\\n\\\\nBut from Bayes theorem:\\\\n\\\\nP( conversion | in the 30-35 age group ) = ( P( in the 30-35 age group | conversion) * P(conversion)) / P(in the 30-35 age group)\\\\n\\\\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?\\\\n"], ["Identify the name of the distribution", "Normal Deviate is the name of the blog; subsampling is one word, not two "], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["
In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users).
\n\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.
\n\nLooking at conversion rate for the segment of users in the 30-35 age group we have:
\n\n- \n
- P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert? \n
- P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion) \n
- P(converting) - overall conversion rate (users who converted / all users) \n
- P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group \n
Intuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)].
\n\nBut from Bayes theorem:
\n\nP(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group)
\n\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?
\n", "I'd like to create a certain kind of voting/recommendation system. I'm sure there must be a name for what I'm trying to do, but I'm not sure.
\n\nBasically, I start with a distribution of binary vectors of length 512. Each vector determines a visual image (that is, an algorithm is applied to the vector to generate a larger image).
\n\nI select 9 vectors from the distribution at random, and display the corresponding images to a user in a 3x3 grid. The user clicks on the one he thinks is \"best.\" After doing this, the distribution should update, and display 9 more images such that the probability of a vector being chosen is directly proportional to its generated image being \"best\".
\n\nAny help on where to look for algorithms/methods to do this kind of thing? I know that I could build up a simple genetic algorithm to do this, but it seems that there might be more direct Bayesian approaches (if I add in some assumptions about how the vectors are related).
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What is the name of this normalization?", "Analysis of participant's results to see who is the most different"], ["I am biologist who is in the process of self-learning of R skills.
\n", "I love to ponder. And that is what I do most of the time.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the total number of comments of all the posts owned by csgillespie?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Does *y* include all inbound `pcs` (divided by total number of hours of operation I'm guessing?) or just those that are handled by the end of the day?", "The asymptotics (and power) depends much more on the total number of positives than the overall total (strictly, the total number of the whichever outcome is rarer). Your total number of positives is 14. That's pretty small. 'Expected value 5 or more in all cells' is only a rule of thumb. \n\nI'm not sure what the small sample adjustment used by epitools is doing exactly (can't find any explanation in the documentation), but it's certainly something more sophisticated than Yates' correction."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["My research project is to come up with a tool for analyzing publication and subscription patterns of Sri Lankan news web sites.\\\\nAs a part of my project, my tool should be capable of finding out the followings, in addition to analyzing the publication patterns of news websites.\\\\n\\\\n\\\\n\u2022 Mostly subscribed news topics (most popular news topics) based on user comments. Usage of user comments to find out the popularity was a requirement of the Supervisor of the project.\\\\n\\\\n\u2022 News topics which increase popularity with respect to time (for example: fewer comments were made on university strike, but all of a sudden the number of comments increases)\\\\n \\\\nAccording to the implementation, my tool extracts news articles and the associated user comments on a daily basis from several news websites. The tool continuously check whether each article will further receive comments by tracking the article for \u201cX\u201d number of days (\u201cX\u201d is a user defined value and can be changed by the user).\\\\n I have collected following statistics for each article that I have extracted from Month April to present.\\\\n\\\\n\u2022 Published date of the article\\\\n\\\\n\u2022 Number of days that the article has received comments\\\\n\\\\n\u2022 Number of comments (excluding replies) made by the readers.\\\\n\\\\n\u2022 Number of replied comments made by the readers.\\\\n\\\\n\u2022 Number of unique readers (nick names) who have made comments and reply comments\\\\n\\\\n\u2022 The entire total number of comments and replies per article\\\\n\\\\n\\\\nIs there a better statistical methodology which I could use for calculating a popularity score for each news article, using some of the factors listed above. Instead of calculating the total number of comments that each article has received and sorting them to select the \u201ck\u201d top most articles.\\\\n", "Hi I have a problem undertanding this:\\\\n\\\\nExample: In a deck of 52 cards, 5 cards are chosen.\\\\n\\\\n What is the probability that all 5 cards have di\ufb00erent face values?\\\\n total number of outcomes = 52 C 5 \\\\n total number of face value combinations = \\\\n 13 C 5 \\\\n total number of suit possibilities, with replacement = 4^5\\\\n\\\\nNow I get the 52 C 5, the total number of possibilities, and I get the 13 C 5 the number of face cards in the deck, so:\\\\n\\\\n P( 5 diff face values) = (13 C 5)(4^5)/(52 C 5)\\\\n\\\\nWhy? What exactly does the 4^5 is doing there? Help.\\\\n"], ["Added the number of obs", "the number of combinations wasnt correct"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["My research project is to come up with a tool for analyzing publication and subscription patterns of Sri Lankan news web sites.\nAs a part of my project, my tool should be capable of finding out the followings, in addition to analyzing the publication patterns of news websites.
\n\n\u2022 Mostly subscribed news topics (most popular news topics) based on user comments. Usage of user comments to find out the popularity was a requirement of the Supervisor of the project.
\n\n\u2022 News topics which increase popularity with respect to time (for example: fewer comments were made on university strike, but all of a sudden the number of comments increases)
\n\nAccording to the implementation, my tool extracts news articles and the associated user comments on a daily basis from several news websites. The tool continuously check whether each article will further receive comments by tracking the article for \u201cX\u201d number of days (\u201cX\u201d is a user defined value and can be changed by the user).\n I have collected following statistics for each article that I have extracted from Month April to present.
\n\n\u2022 Published date of the article
\n\n\u2022 Number of days that the article has received comments
\n\n\u2022 Number of comments (excluding replies) made by the readers.
\n\n\u2022 Number of replied comments made by the readers.
\n\n\u2022 Number of unique readers (nick names) who have made comments and reply comments
\n\n\u2022 The entire total number of comments and replies per article
\n\nIs there a better statistical methodology which I could use for calculating a popularity score for each news article, using some of the factors listed above. Instead of calculating the total number of comments that each article has received and sorting them to select the \u201ck\u201d top most articles.
\n", "On an attempt to solve this problem I've managed to reduce it to finding the expected number of white balls picked until one black ball is observed (let's call that value $v$). Except that, unlike the geometric distribution, this needs to be done without replacement. Hypergeometric distribution doesn't come to the rescue as the number of black balls picked is immaterial (and of course the white balls must be picked consecutively). So I am looking for a way to compute $v$, or at least its underlying distribution. I've tried to search online but couldn't find a distribution that readily fits this.
\n\nSo the problem is:
\n\n\n\n\nConsider an urn having total of $n$ balls, $w$ of them are white, and the rest is black. Pick $j$ balls without replacement. What is the expected number of white balls that will be picked before a black ball is observed ? Notice that the total number of balls picked, $j$, is fixed.
\n
My guess is that cumulative hypergeometric distribution could be used, while dividing somewhere by the number of ways the black balls intersperse the white ones (after the first black ball is observed). But I have little intuition how to go through with this. I.e. to divide before taking the cumulative distribution or after?
\n\nUpdate
\n\nBased on a question in the comments below, to be clear: the desired value is the expected number of white balls observed until either a black ball is observed, or $j$ picks happened, whichever comes earlier.
\n\nThank you.
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What is the confidence interval if I compute the total number of bits based on the average number of bits in sampled bins?", "What is the probability of rolling all faces of a die after n number of rolls"], ["A total newbie in the world of serious programming.
\n", "Video gamer, avid reader, feminist, owned by cats, migraines.
\n\nSome game IDs:
\n\n- \n
- Warcraft/Hearthstone BT Zaralynda#1825 \n
- 3DS 5257-9871-9599 (AC:NL, P:XY) \n
Yesterday I asked this question in which I had 180 subjects with 500 features each. While I was sure that dimensional-reduction is a must in this case (500 features), most of the answers I got said that 500 are not too many.
\n\nSo, My question is: Is there any rule of thumb when one should use dimensional-reduction before the classifier? How many features is too many? (I guess it is depends upon the ratio between the number of subjects and features. Isn't it?)
\n", "I think that what you need is Multiple Correspondence Analysis (MCA). You can lookup the basics on Wikipedia.
\n\nMCA is part of the R-core, in the package MASS. So I suggest you start with
\n\nlibrary(MASS)\n?mca\nHere is the help page for the function mca.
The output of MCA is a set of ordered factors capturing the relationships between your variables. The absolute loadings of the first factor (i.e. the coordinates of the projections of the variables on the first factor) says which groups of variables most heavily influence the variation.
\n\nAs pointed out in the comments, MCA works with categorical and not numeric variables. This means that the ordered relationship (1 < 2 < 3 < ... < 10) will not be taken into account. How well MCA is appropriate depends on the nature of your questions. For instance, if those are binned numeric variables (e.g. monthly income / 1000, rounded up) then I'd say MCA is inappropriate. If these are subjective evaluations (e.g. \"how much does it hurt?\") then I would give it a try because the categories can sometimes be good to show non linear relationships between variables.
\n\nTo do it in practice, in R, you would present your data in a data.frame of 30 columns and as many rows as you have individuals. Each column would be the answers to a question in the form of a factor, meaning that a 9, say, would be interpreted as the category 9, not the score 9.
Say that you have collected this in a data.frame that you call answers you could do the following:
mca_result <- mca(answers)\nplot(mca_results)\nYou would have a biplot representing the individuals, the answers to the questions (as categories) and the relationship between all this expressed by spatial proximity. Two individuals close to each other on that space gave similar answers, two answers close to each other on that space were chosen by the same individuals. Similarly, an individual close to a group of answers has chosen many of them, and an answer close to a group of individuals was chosen by many of them.
\n\nMCA does not natively accomodate block design. A go-around that might be informative is to sum the squares of the loadings of the answers of each block on the first factor. The square of the loading is called the inertia and says how much the variation of a variable (an answer to a question) is represented in the simpler model consisting of only that factor. In R you could do it this way:
\n\ninertia_on_1 <- mca_results$cs[,1]^2\ntapply(X=inertia_on_1, INDEX=rep(1:3, each=100), sum)\nThe part that says INDEX=rep(1:3, each=100) is to build an index corresponding to the variables of each block (labelled 1, 2, and 3) -- and there are 100 of each (because you have 10 categories for 10 questions).
Me
\n\nWeb developer.
\n\nWebsite
\n\n\n\nContact Information
\n\nPlease DON'T contact me for help regarding any answers I provided! If it's suitable for StackOverflow, please post it on StackOverflow. For everything else I can be contacted on tomsotherone [insert the at sign here] googlemail () com
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the display name of the user who last edited the post \"Examples for teaching: Correlation does not mean causation\"?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["correlation", "mean"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["@cardinal - I just copy-pasted from the OP - but the last digits are the user id of the person who copies the link, e.g. my link would be http://math.stackexchange.com/a/73080/3733", "Wikipedia gives the distribution of $r$ for jointly normal variables at http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Use_the_exact_distribution. Correlation does not depend on means or standard deviations, so the distribution of $r$ is the same regardless of the parameters mentioned in the last paragraph."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["Examples for teaching: Correlation does not mean causation", "I'd like to create a certain kind of voting/recommendation system. I'm sure there must be a name for what I'm trying to do, but I'm not sure.\\\\n\\\\nBasically, I start with a distribution of binary vectors of length 512. Each vector determines a visual image (that is, an algorithm is applied to the vector to generate a larger image).\\\\n\\\\nI select 9 vectors from the distribution at random, and display the corresponding images to a user in a 3x3 grid. The user clicks on the one he thinks is \"best.\" After doing this, the distribution should update, and display 9 more images such that the probability of a vector being chosen is directly proportional to its generated image being \"best\".\\\\n\\\\nAny help on where to look for algorithms/methods to do this kind of thing? I know that I could build up a simple genetic algorithm to do this, but it seems that there might be more direct Bayesian approaches (if I add in some assumptions about how the vectors are related)."], ["Identify the name of the distribution", "corrected for the mean in the linear transform"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I'd like to create a certain kind of voting/recommendation system. I'm sure there must be a name for what I'm trying to do, but I'm not sure.
\n\nBasically, I start with a distribution of binary vectors of length 512. Each vector determines a visual image (that is, an algorithm is applied to the vector to generate a larger image).
\n\nI select 9 vectors from the distribution at random, and display the corresponding images to a user in a 3x3 grid. The user clicks on the one he thinks is \"best.\" After doing this, the distribution should update, and display 9 more images such that the probability of a vector being chosen is directly proportional to its generated image being \"best\".
\n\nAny help on where to look for algorithms/methods to do this kind of thing? I know that I could build up a simple genetic algorithm to do this, but it seems that there might be more direct Bayesian approaches (if I add in some assumptions about how the vectors are related).
\n", "In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users).
\n\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.
\n\nLooking at conversion rate for the segment of users in the 30-35 age group we have:
\n\n- \n
- P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert? \n
- P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion) \n
- P(converting) - overall conversion rate (users who converted / all users) \n
- P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group \n
Intuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)].
\n\nBut from Bayes theorem:
\n\nP(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group)
\n\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Examples for teaching: Correlation does not mean causation", "What is the name of the estimator that takes the mean of likelihood?"], ["Developer who does not code and an Architect who does not day-dream !!
\n", "He who asks is a fool for five minutes, but he who does not ask remains a fool forever.
\n\n-- Chinese proverb
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Among the posts owned by csgillespie, how many of them are root posts?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["How far can you get with the suggestion I initially gave? Do you know how to derive the first result? If not, please investigate the posts found by searching our site on [covariance sum](http://stats.stackexchange.com/search?q=[covariance]+sum): many of them show how to manipulate covariances of linear combinations of random variables.", "The procedure is illustrated at http://stats.stackexchange.com/questions/68984, as well as in many more of the thousands of posts found by searching [sampling distribution](http://stats.stackexchange.com/search?q=sampling+distribution). Most of them focus on the sample mean; a few on the sample correlation; and probably none of them describe how to compute the sampling distribution of the SD because (with few exceptions) it is an intractable calculation."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["One crucial question about Mfuzz that remains unanswered is how many clusters to use. I am still confused about it. I am analyzing 20k genes at 5 different timepoints. My best guess is to cluster them by number of GO groupings, but I'm not so sure about that. \\\\n\\\\nBtw, csgillespie, your paper has been a great help!\\\\n\\\\n\\\\nRegards,\\\\n\\\\nShankar", "You're trying to find a \"typical\" deviation from the mean.\\\\n\\\\nThe variance is \"the average squared distance from the mean\". \\\\n\\\\nThe standard deviation is the square root of that.\\\\n\\\\nThat makes it the [root-mean-square](https://en.wikipedia.org/wiki/Root_mean_square) deviation from the mean.\\\\n\\\\ni) Why would we use the average squared deviation? Why is variance interesting? Among other things, because of this [this](http://en.wikipedia.org/wiki/Variance#Sum_of_uncorrelated_variables_.28Bienaym.C3.A9_formula.29) basic fact about variances. That fact is covered in [this question](http://stats.stackexchange.com/questions/41156/algebra-for-data-confidence) among many others.\\\\n\\\\nii) Why take the square root of that? Because then it's in the original units. It measures a particular kind of 'typical distance' from the mean - but because of the above property of variance - one that has some nice features.\\\\n\\\\n"], ["retagged for 'sample-size' you are asking how many to sample", "Clarified that there are many candidates."], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["There are two points here:
\n\n- \n
The passage recommends transforming IVs to linearity only when there is evidence of nonlinearity. Nonlinear relationships among IVs can also cause collinearity and, more centrally, may complicate other relationships. I am not sure I agree with the advice in the book, but it's not silly.
\nCertainly very strong linear relationships can be causes of collinearity, but high correlations are neither necessary nor sufficient to cause problematic collinearity. A good method of diagnosing collinearity is the condition index.
\n
EDIT in response to comment
\n\nCondition indexes are described briefly here as \"square root of the maximum eigenvalue divided by the minimum eigenvalue\". There are quite a few posts here on CV that discuss them and their merits. The seminal texts on them are two books by David Belsley: Conditioning diagnostics and Regression Diagnostics (which has a new edition, 2005, as well).
\n", "There are many time series which are obviously not stationary (I mean, by definition), some of them are even determinstic, yet the unity root tests like ADF/PP test, etc., happily reject the existence of an unity root in these processes.
\n\nSo my question is what kind of methods you used to determine whether a process is a truly stationly stochastic process with reasonable confidence?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Estimating how many individuals are infected", "In a time series study, how many time points are enough for unit root test"], ["The gravatar was made with Mathematica, and inspired by this.
\n\nThe contents of my posts are my own opinion, and do not reflect the opinions of my employer.
\n", "Video gamer, avid reader, feminist, owned by cats, migraines.
\n\nSome game IDs:
\n\n- \n
- Warcraft/Hearthstone BT Zaralynda#1825 \n
- 3DS 5257-9871-9599 (AC:NL, P:XY) \n
I am not a statistics expert but was interested in the validity of the reasoning below.
\n\nSomebody argued that many people had the letter 'J' in their initials. Can someone comment on the validity of the following approach and how it could be improved, notably what probability distribution would a list of firs tnames sorted by popularity follows?
\n\n\n\n
According to wikipedia the most common first names for males in the US according to the 1990 census are, in that order:
\n\n\n\n\n'James','John','Robert','Michael','William','David','Richard','Charles','Joseph',\n and 'Thomas'
\n
Let's use an approximate probability distribution (any better way?) that gives the probabilities to these names according to their order i (starting at 0) following:
\n\n\n\n\nP(name_i) = 0.15 / (i + 2)
\n
This is a probability distribution for at least 1000 names (sums to 1), and starts like this:
\n\n\n\n\n0.0749, 0.0499, 0.0374, 0.0299, 0.0249, ...
\n
meaning that someone has 7.49% chances of being called James. The sum of the probabilities of having a name starting with J from these list is 14% (P(James)+P(John)+P(Joseph)), lets call it Pj. Of course Pj is in fact higher because more names than these 8 in the list of 1000 may start with J.
\n\nNow let's assume a child is given three first names, the probability that at least of them starts with J given pj is 36%!! (from (1-(1-pj)^3)). So every time you meet a new person, you have one third chances to see him having a J on his business card. Let's call this probability pIj, probability of at least one initial starting with a J.
\n\nHowever, when you say overwhelming majority, let's assume that you mean 8 out of 10. The probability that out of 10 people you meet in a row 8 have a name starting with a J is, using binomial distribution, only 0.5% I'm afraid, by comb(10,8)pIj^8(1-pIj)^2 .
\n\nHowever, if you settle with 5 our of 10, your chances rise to 16%.
\n\n\n\n
FYI the original question and post can be found here.
\n", "I'm trying to find more information on how to measure and detect increases in activity in twitter list activity. I have access to all the tweets produced by a list of users. The idea is to try to predict or detect when there is some breaking news that everyone is talking about.
\n\nA sensible thing is to plot number of tweets over time, and proportion of active users in a list (if 10 out of 30 users in a list tweet, proportion of active users = 0.33)
\n\nThe problem with this is that it is difficult to spot spikes in activity due to daily rhythm - people tend to sleep, and only tweet during the day... who knew?
\n\nA quick and dirty example: (Cycle plot?)\nhttp://savedbythegoog.appspot.com/?id=6708dc9a2c90f9f937755542900a1b1957f20f3a
\n\nLooking at a month of data is slightly better - but not really informative:\nhttp://savedbythegoog.appspot.com/?id=014b2246ba3d762b5201f0048812cd0675d5721a
\n\nThose peaks could be annotated with the top hashtags to give some meaning to them.
\n\nThat's what I got so far.
\n\nAre there any specific things i could read up on to get a better measure or make things more interpretable? Any other approaches or ideas more than welcome!
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Advice on finding features of users who converted vs. users who didn't?", "Are ecologists the only ones who didn't know that the arcsine is asinine?"], ["What do I look for in a post? Clarity and Contribution.
\n\nNO vote: If the post is clear, but makes no contribution OR if the post is unclear, but makes a contribution (though I will try to improve the clarity either by comments or direct edit).
\n\nUP vote: If the post is clear and makes a contribution.
\n\nDOWN vote: If the post is unclear and makes no contribution (though I will try to see if I've missed something by commenting).
\n", "I like all kinds of games but mostly strategy and RPGs \nHere is a list of some games I have played and my thoughts http://bronzebeard.wordpress.com/games/
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Among the posts owned by an elder user, how many of them have a score of over 19?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Just a thought. How come you see weekend tasks at work? If your tasks have a time property set by the user, then you can sort them by time and see only the currently relevant ones. If your tasks don't have a time property set by the user, then you are risking never getting them done.", "The procedure is illustrated at http://stats.stackexchange.com/questions/68984, as well as in many more of the thousands of posts found by searching [sampling distribution](http://stats.stackexchange.com/search?q=sampling+distribution). Most of them focus on the sample mean; a few on the sample correlation; and probably none of them describe how to compute the sampling distribution of the SD because (with few exceptions) it is an intractable calculation."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["900 sit-ups a day", "John "]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I have logs for users and posts in a blog platform for 3 years. I can easily find out how many posts each user have made per day/month/year/etc.\\\\n\\\\nWhat I want to find out is if the frequency of posts is growing or not, in other words, if the users are contributing more over time or not. New users could join at any time and old ones could abandon the system (only the first one is logged). Is a trend estimation (http://en.wikipedia.org/wiki/Trend_estimation) suitable for this ? \\\\n\\\\nMore over, is there a python package for this kind of time analyse ? \\\\n", "I have some data examining blog posts on different days. Basically, about 2000 news articles pertaining a certain topic were sampled and each blog post was given a positivity percentage score depending on the amount of positive words in the text (from 0 to 1). \\\\n\\\\nThis data was collected over a week. Therefore, there are 7 separate means, one for each day representing the average positivity of posts for a topic.\\\\n\\\\nI also have a sample of non-topic related news articles over a week that have been coded for positivity as well.\\\\n\\\\nI have a specific hypothesis that positivity in the articles about the topic should have increased on a specific day (e.g. on the 25th). However, this increase should be independent of any sort of non-topic related positivity increase that occurred on that day (i.e. it's not just because all articles are generally happier on that day, it's something specific to the topic).\\\\n\\\\nIs there a way to test that hypothesis with the data collected?"], ["a in title should have been an", "Tried to improve the quality score of the question"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I have some data examining blog posts on different days. Basically, about 2000 news articles pertaining a certain topic were sampled and each blog post was given a positivity percentage score depending on the amount of positive words in the text (from 0 to 1).
\n\nThis data was collected over a week. Therefore, there are 7 separate means, one for each day representing the average positivity of posts for a topic.
\n\nI also have a sample of non-topic related news articles over a week that have been coded for positivity as well.
\n\nI have a specific hypothesis that positivity in the articles about the topic should have increased on a specific day (e.g. on the 25th). However, this increase should be independent of any sort of non-topic related positivity increase that occurred on that day (i.e. it's not just because all articles are generally happier on that day, it's something specific to the topic).
\n\nIs there a way to test that hypothesis with the data collected?
\n", "I have logs for users and posts in a blog platform for 3 years. I can easily find out how many posts each user have made per day/month/year/etc.
\n\nWhat I want to find out is if the frequency of posts is growing or not, in other words, if the users are contributing more over time or not. New users could join at any time and old ones could abandon the system (only the first one is logged). Is a trend estimation (http://en.wikipedia.org/wiki/Trend_estimation) suitable for this ?
\n\nMore over, is there a python package for this kind of time analyse ?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Reducing the number of groups of data by joining them ", "Order of Support Vectors, and how to reduce them"], ["Good ideas want as many people to have them as possible
\n", "MD mesmerized by data science, TeX user, R programmer.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the location of the owner of the post \"Eliciting priors from experts\"?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Sorry, I edited the post so that it has the priors that I meant.", "you get priors for the *parameters* of the priors"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["This is most probably going to be achieved using simulation. Using normal priors for the *log* odds ratios with \"almost 0\" precision, estimating the logistic regression model, and computing the parameters' posterior mode gives you approximate frequentist inference. Increasing this precision can be used to represent the strength of prior information. The power of your Bayesian test will, of course, depend on both the precision and the location of the priors so be certain to state explicitly how this information is used.", "I can't post a comment (the appropriate place for this comment) as I don't have enough reputation, but the answer accepted as the best answer by the question owner misses the point.\\\\n\\\\n\"If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss.\"\\\\n\\\\nThe likelihood is a loss function. Maximising likelihood is the same as minimising a loss function: the deviance, which is just -2 times the log-likelihood function. Similarly finding a least squares solution is about minimising the loss function describing the residual sum of squares. \\\\n\\\\nBoth ML and stats use algorithms to optimise the fit of some function (in the broadest terms) to data. Optimisation necessarily involves minimising some loss function."], ["Formating the post", "clarified the main issue of the post in the title"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["This is most probably going to be achieved using simulation. Using normal priors for the log odds ratios with \"almost 0\" precision, estimating the logistic regression model, and computing the parameters' posterior mode gives you approximate frequentist inference. Increasing this precision can be used to represent the strength of prior information. The power of your Bayesian test will, of course, depend on both the precision and the location of the priors so be certain to state explicitly how this information is used.
\n", "I can't post a comment (the appropriate place for this comment) as I don't have enough reputation, but the answer accepted as the best answer by the question owner misses the point.
\n\n\"If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss.\"
\n\nThe likelihood is a loss function. Maximising likelihood is the same as minimising a loss function: the deviance, which is just -2 times the log-likelihood function. Similarly finding a least squares solution is about minimising the loss function describing the residual sum of squares.
\n\nBoth ML and stats use algorithms to optimise the fit of some function (in the broadest terms) to data. Optimisation necessarily involves minimising some loss function.
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Eliciting priors from experts", "Comparing estimators of location of the Cauchy distribution"], ["I'm a post doc in the area of computational neuroscience.
\n", "What do I look for in a post? Clarity and Contribution.
\n\nNO vote: If the post is clear, but makes no contribution OR if the post is unclear, but makes a contribution (though I will try to improve the clarity either by comments or direct edit).
\n\nUP vote: If the post is clear and makes a contribution.
\n\nDOWN vote: If the post is unclear and makes no contribution (though I will try to see if I've missed something by commenting).
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "From which post is the tag \"bayesian\" excerpted from? Please give the body of the post.", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["bayesian", "Socratic"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Your title seems inconsistent with the body of your post. Do you want the question in the title answered, or the much more specific ones in the body of the post?", "This appears to be a standard textbook-style question; while such questions are welcome here, they are treated differently. Please review the `self-study` [tag wiki info](http://stats.stackexchange.com/tags/self-study/info), and add the tag to your post. With the points in the tag wiki in mind, what have you tried? Where exactly do your difficulties lie?"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["There is a sizable body of work under the title \"contextual bandits\", which might be of interest to you. \\\\n\\\\nA bit more information.\\\\nContextual bandit algorithms attempt to solve the problem of picking the arm of a multi-armed bandit conditioned on some random vector (called the context). There is a discussion of the problem in [this][1] blog post. The traditional application is for internet advertising, where say the system sees a user's query (which is the context) and has to pick from one of K possible ads to show. \\\\n\\\\n\\\\n [1]: http://hunch.net/?p=298", "This tag is ambigous. Please do not use it as is, and use the specific tags [generalized-moments](http://stats.stackexchange.com/tags/generalized-moments/info) for econometric GMM, [gaussian-mixture-model](http://stats.stackexchange.com/tags/gaussian-mixture/info) for machine learning GMM, and [growth-mixture-model](http://stats.stackexchange.com/tags/growth-mixture-model/info) for psychometric GMM.\\\\n\\\\nOne entry is retained with `gmm` tag so that the tag itself does not disappear: http://stats.stackexchange.com/questions/27736/when-should-one-consider-using-gmm. Please do not remove the tag from that entry."], ["Formating the post", "Copied comments into the post."], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["There is a sizable body of work under the title \"contextual bandits\", which might be of interest to you.
\n\nA bit more information.\nContextual bandit algorithms attempt to solve the problem of picking the arm of a multi-armed bandit conditioned on some random vector (called the context). There is a discussion of the problem in this blog post. The traditional application is for internet advertising, where say the system sees a user's query (which is the context) and has to pick from one of K possible ads to show.
\n", "This tag is ambigous. Please do not use it as is, and use the specific tags generalized-moments for econometric GMM, gaussian-mixture-model for machine learning GMM, and growth-mixture-model for psychometric GMM.
\n\nOne entry is retained with gmm tag so that the tag itself does not disappear: When should one consider using GMM?. Please do not remove the tag from that entry.
What do I look for in a post? Clarity and Contribution.
\n\nNO vote: If the post is clear, but makes no contribution OR if the post is unclear, but makes a contribution (though I will try to improve the clarity either by comments or direct edit).
\n\nUP vote: If the post is clear and makes a contribution.
\n\nDOWN vote: If the post is unclear and makes no contribution (though I will try to see if I've missed something by commenting).
\n", "I predict US bank financials and generate the world's largest body of US bank analytics, worldwide.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "From which post is the most popular tag excerpted from? Please give the body of the post.", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Your title seems inconsistent with the body of your post. Do you want the question in the title answered, or the much more specific ones in the body of the post?", "This will answer for one question, for example: \"What are the most popular spices for the most popular vegetables among the most popular fruits, (a) with and (b) without the regard for the color of either?\" For the next question (\"Which color of potatoes is encountered most often along with the most popular vegetable?\") I have to redo the report from scratch \u2014\u00a0unless I miss something."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["There is a sizable body of work under the title \"contextual bandits\", which might be of interest to you. \\\\n\\\\nA bit more information.\\\\nContextual bandit algorithms attempt to solve the problem of picking the arm of a multi-armed bandit conditioned on some random vector (called the context). There is a discussion of the problem in [this][1] blog post. The traditional application is for internet advertising, where say the system sees a user's query (which is the context) and has to pick from one of K possible ads to show. \\\\n\\\\n\\\\n [1]: http://hunch.net/?p=298", "This tag is ambigous. Please do not use it as is, and use the specific tags [generalized-moments](http://stats.stackexchange.com/tags/generalized-moments/info) for econometric GMM, [gaussian-mixture-model](http://stats.stackexchange.com/tags/gaussian-mixture/info) for machine learning GMM, and [growth-mixture-model](http://stats.stackexchange.com/tags/growth-mixture-model/info) for psychometric GMM.\\\\n\\\\nOne entry is retained with `gmm` tag so that the tag itself does not disappear: http://stats.stackexchange.com/questions/27736/when-should-one-consider-using-gmm. Please do not remove the tag from that entry."], ["sampling programs in the three most popular stats packages", "Formating the post"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["There is a sizable body of work under the title \"contextual bandits\", which might be of interest to you.
\n\nA bit more information.\nContextual bandit algorithms attempt to solve the problem of picking the arm of a multi-armed bandit conditioned on some random vector (called the context). There is a discussion of the problem in this blog post. The traditional application is for internet advertising, where say the system sees a user's query (which is the context) and has to pick from one of K possible ads to show.
\n", "The following is my experience doing some researches linked to statistical concepts. In the most situations I had no idea on how to organize the questions, how to ask for answers, and how to provide feedback showing my acceptance or rejection. I'm certainly not talking about the language issue but pointing on that it was not clear to me what the concept behind my question is. Sometime it comes out that there is a misunderstanding regarding the question. Some people do not tolerate the lack of formality (~standards) in the question and so their replies are not useful but discouraging. I believe that there is some sort of kindness and helpfulness trying to answer questioner with the hope being satisfied.
\nProvided this background, I would appreciate it if you would provide me a list of: tools / tips-and-tricks / methods / skills / techniques / strategies / ... to find out how to seek for solution / help / ... in facing with statistical problems.
\nFor example this site (Stats.StackExchange.com) is a tool providing a forum based technology to ask statistical questions. Some people probably recommend reading books etc which I consider them not a shortcut solution although for young scholars it could be a good idea. The reason is the questioner in my question has lots of thing to do and also has other profession.
\nMy experiences showed that asking a clear/complicated question from an academic (unknown/famous) who does not know you has no way to be succeed. For a keen researcher it is however disappointing.
\nFor this question I tried to be clear as I could if you see it not clear enough, you got the problem. Any ideas?
Please read this before commenting:
\nI accept that the main question (the idea behind this post) might be a bit hard to be caught up by immediate-commentators. Briefly, in the first time facing with a problem we would have some guesses about the related topics/subject/methods in statistics that could help us to understand the problem and to propose solutions. This post discusses the problem solving strategies to connect a proper statistical analysis to the problem being investigated even roughly.
Although productivity could be developed by the help of professional statistician however this is not the case to be addressed in this post. The main purpose of this post is to help self-learning practitioners who have enough background about the main concepts being investigated however do not have a clear idea if there is any modern/classic developed concept regarding the problem or not. For statistics as for other parts of sciences there are always tips-and-tricks that could help the researcher to speed up the progress of productivity, at least I have a bunch of them.
I would recommend commentators to give a fair opportunity for discussions even if it is not their favorite. Please think positive and deep you'll see good points in this post. Nevertheless, to me what you comment or mark or vote will be useful even if those are not directly addressing the question and regardless the tone/language all are appreciated.
\nComments-on-comments
\n@cardinal: Thanks. the provided link is an interesting post.
What do I look for in a post? Clarity and Contribution.
\n\nNO vote: If the post is clear, but makes no contribution OR if the post is unclear, but makes a contribution (though I will try to improve the clarity either by comments or direct edit).
\n\nUP vote: If the post is clear and makes a contribution.
\n\nDOWN vote: If the post is unclear and makes no contribution (though I will try to see if I've missed something by commenting).
\n", "I predict US bank financials and generate the world's largest body of US bank analytics, worldwide.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "How many badges has the user csgillespie obtained?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["How are these values from the 'theoretical' distributions obtained?", "How were the estimates obtained? Do you know the sample size for each? Do you know the the shape parameter of these gammas or not?"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.\\\\n\\\\nCurrently, I've filtered the raw data down to a set of:\\\\n\\\\n - User ID\\\\n - Year and Month\\\\n - Posts made during that month\\\\n - Total badges that user has earned up to that month\\\\n\\\\nI'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.\\\\n\\\\nAny ideas?\\\\n\\\\nThank you!", "I have a dataset with page view data for about 500,000 users, divided into two groups. Each user can visit up to 5 pages, each as many or as few times as they want. So for each user, I have the distribution of number of visits to each page. I would like to compare the 'average skew' in distribution between the two groups. Roughly, users in one group are more likely to have distributions that look like this {3,0,0,0,0} and users in the other group are more likely to have distributions like this {1,1,0,1,0}. How can I compare the degree of skewness between the two groups? I thought of using the average Gini coefficient or entropy for each group, but each user has so few observations. How can I do this?"], ["How to enter data.", "many spelling errors"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.
\n\nCurrently, I've filtered the raw data down to a set of:
\n\n- \n
- User ID \n
- Year and Month \n
- Posts made during that month \n
- Total badges that user has earned up to that month \n
I'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.
\n\nAny ideas?
\n\nThank you!
\n", "I have a dataset with page view data for about 500,000 users, divided into two groups. Each user can visit up to 5 pages, each as many or as few times as they want. So for each user, I have the distribution of number of visits to each page. I would like to compare the 'average skew' in distribution between the two groups. Roughly, users in one group are more likely to have distributions that look like this {3,0,0,0,0} and users in the other group are more likely to have distributions like this {1,1,0,1,0}. How can I compare the degree of skewness between the two groups? I thought of using the average Gini coefficient or entropy for each group, but each user has so few observations. How can I do this?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["How many user interactions do I need to perform search relevance experiment?", "How are user opinions compared?"], ["A once do-it-all developer has turned the corner to focus mainly on user experience, visual design, and user interaction. I'm a very technically adept developer, who has decided to hone in on the visual design/interaction side of web applications, and iOS development.
\n", "Ubuntu user since the end of 2008.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Please list the names of the badges obtained by csgillespie.", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Please list the x and y values for the second peak in each graph.", "That is correct (see my recent edit). I have a list of full names where the order is inconsistent."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.\\\\n\\\\nCurrently, I've filtered the raw data down to a set of:\\\\n\\\\n - User ID\\\\n - Year and Month\\\\n - Posts made during that month\\\\n - Total badges that user has earned up to that month\\\\n\\\\nI'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.\\\\n\\\\nAny ideas?\\\\n\\\\nThank you!", "It seems that `names <- ls(ini$imp)` sorts the naming of the list `ini$imp` alphabetically. Changing the names vector to `names <- names(ini$imp)` should fix this. We will fix this in the next version of `mice`. Thanks. "], ["improve the list", "formatted the numbered list"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I'm trying some metrics to filter a list of gene names with associated numerical values \nwhich correspond to their abundance.
\n\nI'm looking for some interesting genes and trying to see if they come up to the top region of the ranked gene list (ranked by abundance)\nbefore and after filtering. Please see Fig. enclosed.\n![\"enter](\"http://i.stack.imgur.com/rCihr.png\")
I wish to see if the ranking of my gene of interest (G) has changed significantly before and after filtering and I'm thinking\nto do a Fisher's Exact test for proportions to ascertain the significance.
\n\nSo given the rank of the gene of interest I'm willing to look at the number of genes above and below the gene in rank \nwith and without the filtering and compare their proportions using FE Test
\n\nIf in the UN-Filtered data there are 200 and 2800 genes above and below G and 30 and 100 in the Filtered data respectively\nthen I can do the FE test as follows (in R):
\n\n>my.mat <- matrix(c(30,100,200,2800),nrow=2,byrow=TRUE)\n>my.mat\n [,1] [,2]\n[1,] 30 100\n[2,] 200 2800\n\n>fisher.test(my.mat,alternative=\"two.sided\") \nPlease let me know whether I can do a Fisher Exact test AT ALL for such kind of measurement ?
\n\nThanks in advance
\n", "I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.
\n\nCurrently, I've filtered the raw data down to a set of:
\n\n- \n
- User ID \n
- Year and Month \n
- Posts made during that month \n
- Total badges that user has earned up to that month \n
I'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.
\n\nAny ideas?
\n\nThank you!
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What is the estimation bias of the top estimate in a list sorted by value?", "Test the significant differences between the coefficients obtained by logistic regression on two data sets."], ["I like studying the zeros of functions. Here's a list of my answers on the topic here on Math.SE. I also enjoy answering questions about asymptotics in general.
\n\nMy avatar is cropped from this image by Garyck Arntzen.
\n", "Currently a graduate student in mathematics at the University of California - Berkeley.
\n\nPreviously obtained a MASt. in Applied Mathematics from the University of Cambridge (2013), and a B.S. in Mathematics and a B.A. in Physics from the University of Chicago (2012).
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Among the badges obtained by csgillespie, how many of them were obtained in the year 2011?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["P.S., we did not design the studies, the results were obtained from the literature.", "The standard errors are obtained by taking the diagonal elements of the inverse X'X matrix and multiplying them by the estimated standard deviation of the errors."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I have seen many papers using 10-fold CV (\"10-pool\" CV),\\\\nbut I think the accuracy obtained this way can be,\\\\nsometimes, optimistically incorrect\\\\nsince at any given time step **t** the sum of distinct queries of all folds\\\\nmight be greater than **t**.\\\\n\\\\n**t** is normally used in plots as the number of queries (seen instances).\\\\n\\\\nAnother problem is the impossibility to know how many queries were used, expect by averaging across the folds.\\\\n\\\\nAnother approach commonly used is the Holdout 50/50, 70/30 etc.\\\\nHold-out does not have those specific problems, but is inadequate to small datasets.", "I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.\\\\n\\\\nCurrently, I've filtered the raw data down to a set of:\\\\n\\\\n - User ID\\\\n - Year and Month\\\\n - Posts made during that month\\\\n - Total badges that user has earned up to that month\\\\n\\\\nI'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.\\\\n\\\\nAny ideas?\\\\n\\\\nThank you!"], ["Added explanation of how numbers were obtained", "Explained how the values in the plots were obtained."], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.
\n\nCurrently, I've filtered the raw data down to a set of:
\n\n- \n
- User ID \n
- Year and Month \n
- Posts made during that month \n
- Total badges that user has earned up to that month \n
I'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.
\n\nAny ideas?
\n\nThank you!
\n", "I have location data from 5 years (>30,000 points). Each location is given a classification name (in my case vegetation classes). These vegetation classes were assigned by intersecting the locations with vegetation maps representing 5 different years. The vegetation categories changed due to clear cut practices. Basically, some old forest areas in 2007 became clear cut in 2008, and others in 2009 became clear cuts in 2010 etc. Also, some old clear cuts became young forests because they grew back. What I want to know is, if I used just 1 map (say 2007) - how many wrong classifications will I have per year after this? Because what was once forest in 2007 could be clear cut in 2009. Knowing that the location from year 2009 is a clear cut, using the map from 2007 - was it misclassified? I will want to test this for each year. Basically, is there a map that minimizes miss-classification of points.
\n\nIn the data set for each XY location, I have Year location was collected, clearcut07, clearcut08, clearcut09..., Names07, Names08, Names09....
\n\nThe clearcut07 is binary indicator of 1 or 0, if was really a clear cut that year. the Names07 are the categorical names for all locations (all 5 years of points - if they were to be classified from that year). So, all locations obtained are given the attributes from the 2007 map, then the 2008 map then 2009, etc.
\n\nI need a summary of counts of locations within each category. A count of locations that were properly classified as clear cuts from each year of the data. A count of locations from data collected, for example, in 2008 that were classified as clear cuts from each map? And a count of locations that were incorrectly classified as clear cuts from each map when I know the correct classification.
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["How to compare results obtained by non-linear regression", "Validity of PCA obtained results"], ["Currently a graduate student in mathematics at the University of California - Berkeley.
\n\nPreviously obtained a MASt. in Applied Mathematics from the University of Cambridge (2013), and a B.S. in Mathematics and a B.A. in Physics from the University of Chicago (2012).
\n", "Computer scientist, mathematician, Go player, amateur musician. I obtained a PhD in Artificial Intelligence in 2009. Currently I work as a post-doc.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the display name of the user who has obtained the most number of badges?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["@David For the record, Akavall is the one who suggested predicting 1 (since it's the number that occurs most often)", "Thank you Pete for your answer, it made the problem clearer. I do not understand why we use predictions[u][m] in the for loop though. Why is the test user in the generated user factor matrix? Do we evaluate the algorithm using the factor matrices obtained from the whole dataset? I am considering the case where I split the matrix so that the training matrix has 95% of the users(row-wise) and the rest of the users are in the testing matrix."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["The Wikipedia article has accurate information about A/B testing; binary hypothesis testing is another name for A/B testing. A/B testing and split testing are the most widely accepted terms in the business and marketing community. The exact origins of A/B testing are not well known but can be traced back to [Google during the turn of the millennium][1]. \"Google engineers ran their first A/B test at the turn of the millennium to determine the optimum number of results to display on a search engine results page.\"\\\\n\\\\n\\\\n [1]: https://www.pardot.com/blog/abcs-ab-testing/", "In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users).\\\\n\\\\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.\\\\n\\\\nLooking at conversion rate for the segment of users in the 30-35 age group we have:\\\\n\\\\n* P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert?\\\\n* P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion)\\\\n* P(converting) - overall conversion rate (users who converted / all users)\\\\n* P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group\\\\n\\\\nIntuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)].\\\\n\\\\nBut from Bayes theorem:\\\\n\\\\nP(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group)\\\\n\\\\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?"], ["Identify the name of the distribution", "name of the process"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users).
\n\nIf we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group.
\n\nLooking at conversion rate for the segment of users in the 30-35 age group we have:
\n\n- \n
- P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert? \n
- P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion) \n
- P(converting) - overall conversion rate (users who converted / all users) \n
- P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group \n
Intuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)].
\n\nBut from Bayes theorem:
\n\nP(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group)
\n\nIs this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?
\n", "The Wikipedia article has accurate information about A/B testing; binary hypothesis testing is another name for A/B testing. A/B testing and split testing are the most widely accepted terms in the business and marketing community. The exact origins of A/B testing are not well known but can be traced back to Google during the turn of the millennium. \"Google engineers ran their first A/B test at the turn of the millennium to determine the optimum number of results to display on a search engine results page.\"
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Analysis of participant's results to see who is the most different", "What is the name of this normalization?"], ["A once do-it-all developer has turned the corner to focus mainly on user experience, visual design, and user interaction. I'm a very technically adept developer, who has decided to hone in on the visual design/interaction side of web applications, and iOS development.
\n", "You can find my display name on twitter..."], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the average score of the posts owned by the user csgillespie?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["What I mean by fraud is that a user that follows the strategy of assigning 50 to each of the 13 items will likely always end up with the highest score because it would be very difficult for genuine users who are rating the items correctly to beat this 336 score in most cases.", "I'm actually looking to assign a score to each classification, so that I can use the score to order the documents. For example, it would be useful in a recommendations setting where I want to classify documents into those that the user will like and those that the user will dislike, then use the classification score to order the recommendations."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I'm in the process of designing a system. \\\\n\\\\nSo basically, every day users on my site rack up points based on the number of times they visit my website. \\\\n\\\\nSo on 8/3, if a user visited the site 10 times, they have 10 points for that day.\\\\n\\\\nEvery day, I figure out the number of times the user visited my site (points) for that day and add a row in the database with the number of points for that day.\\\\n\\\\nI want to give each person a score. Basically a score from 0 to 10 just like in school, but I want that score to depend on the mean score of all the users. \\\\n\\\\nI was thinking of finding the average number of points per day for each user (so user 1 might have an average of 10 points per day) and then finding the average ($X$) of these averages. And using $X$ to create a normal distribution to give each person a score from 0 to 10. \\\\n\\\\nBut if the person is 2 standard deviations from the mean, how does that translate to a score?\\\\n\\\\nI'm not a statistician. Any advice would be greatly appreciated. What's the best way to calculate a score like this? Also I know with normal distributions that the distribution has to be normal. What if the average scores are right skewed? What should I do? Determine whether there is an outlier? Anyway help would be appreciated.", "I'm in the process of designing a system. \\\\n\\\\nSo basically, every day users on my site rack up points based on the number of times they visit my website. \\\\n\\\\nSo on 8/3, if a user visited the site 10 times, they have 10 points for that day.\\\\n\\\\nEvery day, I figure out the number of times the user visited my site (points) for that day and add a row in the database with the number of points for that day.\\\\n\\\\nI want to give each person a score. Basically a score from 0 to 10 just like in school, but I want that score to depend on the mean score of all the users. \\\\n\\\\nI was thinking of finding the average number of points per day for each user (so user 1 might have an average of 10 points per day) and then finding the average (X) of these averages. And using X to create a normal distribution to give each person a score from 0 to 10. \\\\n\\\\nBut if the person is 2 standard deviations from the mean, how does that translate to a score?\\\\n\\\\nI'm not a statistician. Any advice would be greatly appreciated. What's the best way to calculate a score like this? Also I know with normal distributions that the distribution has to be normal. What if the average scores are right skewed? What should I do? Determine whether there is an outlier? Anyway help would be appreciated."], ["Tried to improve the quality score of the question", "Added more information on the average of the field, as requested in the comments."], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I'm in the process of designing a system.
\n\nSo basically, every day users on my site rack up points based on their activity on my website. So whatever they do on my website, they rack up points.
\n\nSo on 8/3, a user might do multiple things on my website so that they rack up 10 points.
\n\nEvery day, I figure out the number of points for the user for that day and add a row in the database with the number of points for that day.
\n\nI want to give each person a score. Basically a score from 0 to 10 just like in school, but I want that score to depend on the mean score of all the users.
\n\nI was thinking of finding the average number of points per day for each user (so user 1 might have an average of 10 points per day) and then finding the average ($X$) of these averages. And using $X$ to create a normal distribution to give each person a score from 0 to 10.
\n\nBut if the person is 2 standard deviations from the mean, how does that translate to a score?
\n\nI'm not a statistician. Any advice would be greatly appreciated. What's the best way to calculate a score like this? Also I know with normal distributions that the distribution has to be normal. What if the average scores are right skewed? What should I do? Determine whether there is an outlier? Anyway help would be appreciated.
\n", "I suggest to use Expected Utility or R-score. Assume your model has created an ordered list of recommendations where the first item is the one the user is most likely and the last is the one the user is least likely interested in. Let's say that this recommendations are specified by $r_i$ where $i$ is the position in the list.
\n\nExpected utility for a particular user u is now defined as:\n$R_u=\\\\sum_{i=1}^{n}\\\\frac{f_u(r_i)}{2^{\\\\frac{i-1}{\\\\alpha-1}}}$
\n\nwhere ...
\n\n- \n
- $f_u(r_i)=1$, if the recommended item $r_i$ is the user's library, else 0 \n
- $\\\\alpha$ specifies the slope of the decay. The smaller alpha, the greater the slope. \n
This metric measures how likely it is that a user will view a recommended item, assuming a list of items is recommended to him. If the item of interest is placed at a very high position, it is unlikely that the user will bother to scroll/look that far, even if it exactly what he/she wanted. In this sense, $\\\\alpha$ specifies how patient the average user is.
\n\nTo calculate the R-score for a set of users, it is recommended to normalize the R-score per user beforehand. The resulting score is
\n\n$R =\\\\sum_u\\\\frac{R_u}{R_u^*}$
\n\nwhere $R_u^*$ is the maximum R-score you can achieve for one user. That is, given a user has k items in his library you want to predict, the items at the first k positions are exactly these ones.
\n\nFor more metrics or as a general read I recommend Evaluating Recommendations Systems by Shani & Gunarwadana
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What is the intuition behind the score function?", "What is the training score used for?"], ["Video gamer, avid reader, feminist, owned by cats, migraines.
\n\nSome game IDs:
\n\n- \n
- Warcraft/Hearthstone BT Zaralynda#1825 \n
- 3DS 5257-9871-9599 (AC:NL, P:XY) \n
Ubuntu user since the end of 2008.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the average number of badges obtained by a user with over 200 views?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[200, 30325], [200, 14185], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[200, 67584], [200, 32224], [90, 57], ["JMP software uses a better method. It gives the user a default number of bins but the user can then drag over the plot to increase or decrease the number of bins. That lets him (or her) see it change in real time.", "But the average over all $k$ and $n$ folds, respectively averages the same number of cases..."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [200, 8958], ["900 sit-ups a day", "This is much healthier."]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[200, 118983], [38, 37], [200, 34289], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [200, 16472], ["I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.\\\\n\\\\nCurrently, I've filtered the raw data down to a set of:\\\\n\\\\n - User ID\\\\n - Year and Month\\\\n - Posts made during that month\\\\n - Total badges that user has earned up to that month\\\\n\\\\nI'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.\\\\n\\\\nAny ideas?\\\\n\\\\nThank you!", "As @MichaelChernick says, there is probably no good answer to reliably estimate the user's loyalty, for users who have not used any service very often.\\\\n\\\\n(Arguably it would be reasonable to classify those users as not very loyal. You might decide that a user must us your service a minimum number of times before you're prepared to consider them a loyal user.)\\\\n\\\\nOne approach to dealing with these issues might be to use a Bayesian method. Compute the average loyalty, across all users. For a user who has no uses, the average loyalty is your best estimate. As the number of uses increases, your estimate should start to move in the direction of your $L(A)$ value.\\\\n\\\\nIn particular, the following might be a reasonable estimate:\\\\n$$L_\\\\text{est}(A) = \\\\frac{n}{n+n^*} L(A) + \\\\frac{n^*}{n+n^*} L^*(A), $$\\\\nwhere $n=N(A)+N(B)$ is the number of times this user has used a service, $n^*$ is the average value of $n$ across all users, $L(A)=N(A)/(N(A)+N(B))$ is your naive (uncorrected) loyalty ratio for this user, and $L^*(A)$ is the average value of $L(A)$ across all users.\\\\n\\\\nIf you know something about the characteristics of the user, and you think these characteristics will have a significant effect on user loyalty, you could perhaps take the average over all other users who are similar to this user, rather than over all users."], ["200", "$\\\\LaTeX$; user tagged with @"], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[200, 34285], [7, 6], [200, 10197], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [200, 1257], ["I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.
\n\nCurrently, I've filtered the raw data down to a set of:
\n\n- \n
- User ID \n
- Year and Month \n
- Posts made during that month \n
- Total badges that user has earned up to that month \n
I'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.
\n\nAny ideas?
\n\nThank you!
\n", "As @MichaelChernick says, there is probably no good answer to reliably estimate the user's loyalty, for users who have not used any service very often.
\n\n(Arguably it would be reasonable to classify those users as not very loyal. You might decide that a user must use your service a minimum number of times before you're prepared to consider them a loyal user.)
\n\nOne approach to dealing with these issues might be to use a Bayesian method. Compute the average loyalty, across all users. For a user who has no uses, the average loyalty is your best estimate. As the number of uses increases, your estimate should start to move in the direction of your $L(A)$ value.
\n\nIn particular, the following might be a reasonable estimate:\n$$L_\\\\text{est}(A) = \\\\frac{n}{n+n^*} L(A) + \\\\frac{n^*}{n+n^*} L^*(A), $$\nwhere $n=N(A)+N(B)$ is the number of times this user has used a service, $n^*$ is the average value of $n$ across all users, $L(A)=N(A)/(N(A)+N(B))$ is your naive (uncorrected) loyalty ratio for this user, and $L^*(A)$ is the average value of $L(A)$ across all users.
\n\nIf you know something about the characteristics of the user, and you think these characteristics will have a significant effect on user loyalty, you could perhaps take the average over all other users who are similar to this user, rather than over all users.
\n"], [200, 16481], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What's the best formula to fit the distribution of website user number over a day", "What is a tertile average of a group?"], ["An ubuntu user learning the ropes of systems administration. Currently watches over 7 ubuntu boxes.
\n", "Software developer, business problem solver, husband, dad, and under the finance IT sector challenges!\n
![\"The](\"http://ubuntucounter.geekosophical.net/img/ubuntu-blogger.php?user=35232\")
I am comparing the Facebook posts of a TV news organization's Facebook Page to the contents of the TV news broadcast. (This is called \"agenda setting\" in communications research.) I have used a grounded theory approach to code hundreds of Facebook posts and segments of the TV show.
\n\nHere is the problem: There are about 1500 Facebook posts with 500 code categories and 51 TV segments with 300 code categories. When I test the correlation, I have been testing the top 20 Facebook post categories against the number of times the same categories appear on the TV show. Some people have suggested that I need to test all of the Facebook categories against all of the TV show categories. Is this true? They are not equal in number. As a compromise I have converted the number of times a code appears in Facebook or on TV to percentage - percentage out of 1500 for Facebook and percentage out of 51 for the TV show. Does this seem reasonable? Is there a better way to do this without comparing the 500 to the 300 codes. I am using Pearson's correlation coefficient.
\n", "I'm trying to get a measure for a set of data that indicates someone's \"spam score\". Essentially, the higher the spam score, the more likely they are to be spammers.
\n\nRight now, I'm measuring a person's spam score as a ratio: bad posts/total posts. If this ratio is high, then they're likely to be spammers.
\n\nThe difficulty I'm having is trying to compare different people - for example, a person with 6/8 bad posts is not AS likely to be as spammy as someone who has 600/800 bad posts (the person who's made 600 bad posts is clearly a spammer, but the one who's made 6 has not proven themselves to be a spammer to that extent).
\n\nHowever, right now they are being assigned the same spam score of 6/8 = 0.75. Is there a way I can account for the size of the sample in my spam score?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Comparing noisy data sequences to estimate the likelihood of them being produced by different instances of an identical Markov process", "Probability of what is being predicted in glm"], ["...currently trying to give a little back to SO after using it extensively to get started with Android and other Java programming (and marvelling at the ability of SE sites to aid human progress and organize knowledge). (I made a few posts a year ago, but really just got started.)
\n\nAt the moment:
\n\n- \n
on SO (and SU a bit) this means answering what few questions I can.
\non Meta-SO this means using my fresh perspective to provide feedback on what about the site can violate new users' expectations and confuse them (AKA being a whiny n00b).
\n
The gravatar was made with Mathematica, and inspired by this.
\n\nThe contents of my posts are my own opinion, and do not reflect the opinions of my employer.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "How many votes did the user No.58 take on 2010/7/19?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["\"the user remembers his/her actions, and decides on the next action based on the complete history after entering the system\" How did you assess this?", "How many objects will be in the set? How many objects will receive votes, how many will receive a considerable amount of votes? Are votes eternal, or can they be retracted or changed?"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I have developed a web application in which I show a picture and two captions. One of those captions is the correct and the other is randomly selected from a corpus of picture captions.\\\\n\\\\nNext, I present the following question: \"Which caption best describes the picture?\" and the user casts a vote on the caption that he chooses.\\\\n\\\\nSince I store the user votes, I know how many votes each picture received and also, the fraction of votes that were on the correct caption.\\\\n\\\\nSince the 'correct' captions of some pictures are misleading, it is expected that, in these cases, the correct vote fraction is near %50 (that is, it is impossible to identify the correct caption). However, some pictures will have very descriptive captions, and the correct vote fraction is expected to be somewhat higher.\\\\n\\\\nIs it possible, given a confidence value (e.g. 95%), to identify pictures whose correct caption is impossible for the users to identify (correct vote fraction = %50)?\\\\n", "I'm developing an application in which users can create 'sections' (\u00e0 la subreddit in reddit), in which items/posts can be created and voted with a thumbs-up/down system.\\\\n\\\\n[A great article][1] guided me on how to sort these votes so that an item with a 100% positive response but with few votes won't get ranked over one with hundreds of votes and an acceptance of 80%. The article describes it pretty well.\\\\n\\\\nHowever, I'd like to discard the lowest-ranked items and this is where it gets tricky:\\\\n\\\\n- How could I know the minimum number of votes in order to discard it?\\\\n- What is the score's threshold required to discard the item?\\\\n\\\\nAs I said, there are sections, and each one has items (which are the ones voted). The formula has to take into consideration the fact that one section may have 100 items with thousands of votes and another might have 3 or 4 items with 20 votes, so a minimum of 40 votes required might be optimal for the first case but totally out of bounds for the second.\\\\n\\\\n(I was tempted in posting this to MathOverflow, but I'm not really sure since this also involves some programming)\\\\n\\\\nThanks!\\\\n\\\\n\\\\n [1]: http://www.evanmiller.org/how-not-to-sort-by-average-rating.html"], ["Improved the question based on user comments", "How to enter data."], ["User", "user38406"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I have developed a web application in which I show a picture and two captions. One of those captions is the correct and the other is randomly selected from a corpus of picture captions.
\n\nNext, I present the following question: \"Which caption best describes the picture?\" and the user casts a vote on the caption that he chooses.
\n\nSince I store the user votes, I know how many votes each picture received and also, the fraction of votes that were on the correct caption.
\n\nSince the 'correct' captions of some pictures are misleading, it is expected that, in these cases, the correct vote fraction is near %50 (that is, it is impossible to identify the correct caption). However, some pictures will have very descriptive captions, and the correct vote fraction is expected to be somewhat higher.
\n\nIs it possible, given a confidence value (e.g. 95%), to identify pictures whose correct caption is impossible for the users to identify (correct vote fraction = %50)?
\n", "I've generated a user test to compare two methods: M1 and M2. I generate 40 test cases and show the result of each method on test case to 20 individuals, side by side, the individuals don't know what result came from which method. For each test case each person has to say if the result computed by M1 is better or M2 is better or they are equally good.
\n\nI want to know if M1 is better than M2. I add up all the results and generate 3-D histogram, votes for M1, votes for tie, and votes for M2.
\n\nIf I only looked at M1 and M2 as 2-D histogram. I know that if M1 and M2 were equally good this histogram would be uniform. Then I'll just perform $\\\\chi^2$ test.
\n\nWhat I don't know how to model are the votes for tie. Here are two options I've thought of:
\n\n- \n
- The basis of chi-squared test is that histograms are mutually\nexclusive and add up to one. It seems like the votes for tie can be\nsplit in two and added to each M1 and M2 (and ties removed), but this\ndoes not seem very principled. \n
- Another option is that I could just ignore the ties, that seems\nflawed because it breaks the \"add up to one\" property. For example if\nI had (M1:2, ties:98 M2:0) the difference between both methods would\nbe not statistically significant. \n
What else can I do? Am I looking at this incorrectly? This seems like a common problem people would face when modeling user votes. What is correct way to model the ties?
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Predicting user selections based on similar user", "How did Efron imagine the bootstrap?"], ["A boy swallows a whole jar of coins, and is taken to a hospital. When the\n doctor came out to speak with the parents, he said, \"No change yet.\"
\n\n\n\n
Outside a dog a book is man's best friend. Inside a dog, it's hard to read.
\n\n\n\n
A man needed major surgery. They removed all the organs on the left side of his body.
\n\nNow he is all right. :)
\n\n\n\n
Q: Why did the farmer watch his lazy cows?
\n\nA: He liked to see the meat loaf.
\n\nQ: Why did the farmer feed sugar to his sick pig?
\n\nA: He wanted sugar-cured ham.
\n\n\n\n
Q: How did Luke Skywalker know what Darth Vader gave him for Christmas?
\n\nA: He felt his presents.
\n", "At various times a user of python (esp. django, tornado, and numpy/scipy), javascript (esp. jquery), Cognos, and statistical analysis packages similar to the programming language R. Also did a few years in a mainframe gig.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Indicate the creation date of the maximum number of votes.", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["Is the max number of users an imposed maximum or the maximum observed?", "Do you have a prior on the distribution of the number of clusters? If so you could just pick the number of clusters with the maximum likelihood."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I am dealing with a set of surgery patient data, have the date of surgery, date of discharge, and date of the last follow-up visit.\\\\n\\\\nI am wondering that for calculating the duration of follow-up, do I take the date difference between the last follow-up visit date and the date of discharge, or the last follow-up date and the date of surgery?\\\\n\\\\nThank you!", "I have a set of dataset:\\\\n\\\\n- X variable is date (from April to October)\\\\n- Y variable is vegetation biomass data\\\\n\\\\nIn my study area, growing season start around April when vegetation biomass is low and peak around at the end of August when biomass is highest and fininsh around October. \\\\n\\\\nThe purpose is to determine the exactly date when the vegetation biomass increased maximum during the period of growing season start.\\\\nIt should be in April. \\\\n\\\\nFirst, I did the curve fitting using Sigmoid, logistic, 4 parameter method which was the best fitted for this dataset. And I got a formula of sigmoid, logistic 4 parameter method.\\\\n\\\\nNow, how can I calculate maximum increase date of vegetation biomass from curve fitting line to accomplish the purpose as mentioned above?\\\\n\\\\nThanks a lot. "], ["added the creation of data", "corrected the date of MW!"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I have a dataset:
\n\n- \n
- X variable is date (from April to October) \n
- Y variable is vegetation biomass data \n
In my study area, growing season starts around April when vegetation biomass is low and peaks around at the end of August when biomass is highest, and finishes around October.
\n\nThe purpose is to determine the exactly date when the vegetation biomass increased at maximum during the start of growing season.\nIt should be in April.
\n\nFirst, I did the curve fitting using Sigmoid, logistic, 4 parameter method which was the best fit for this dataset. And I got a formula of sigmoid, logistic 4 parameter method.
\n\nNow, how can I calculate maximum increase date of vegetation biomass from curve fitting line to accomplish the purpose as mentioned above?
\n\nThanks a lot.
\n", "I am dealing with a set of surgery patient data, have the date of surgery, date of discharge, and date of the last follow-up visit.
\n\nI am wondering that for calculating the duration of follow-up, do I take the date difference between the last follow-up visit date and the date of discharge, or the last follow-up date and the date of surgery?
\n\nThank you!
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["Extracting metadata (author, creation date ...) from word documents with R", "Estimating/predicting maximum of sum process for modeling number of cars on the road?"], ["I've worked in the finance sector for the last seven years, using a number of different technology stacks.
\n", "Graduate student in mathematics at the University of Chicago.
\n\nMy interests include algebraic geometry and algebraic number theory.
\n\nHere are some of my favorite quotations.
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "Give the number of \"Revival\" badges.", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Revival", "Socratic"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["What is the number of dimensions of your dataset? What statistical model are you fitting? (an outlier is defined wrt to a model). Give more details. As formulated the question doesn't make sense.", "Give a better reference to the article you cite."], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["A standard example is the [Monty-Hall][1] game. \\\\n\\\\nHere's how I approach this example:\\\\n\\\\n * Give the class sets of three cards and get them to play the game in pairs.\\\\n * Each pair plays the game following a particular strategy, i.e. always switching doors.\\\\n * Afterwards, I use the number of times the class won to calculate a Monte-Carlo estimate of winning. \\\\n\\\\n\\\\n [1]: http://en.wikipedia.org/wiki/Monty_Hall_problem", "I clustered data according to latitude and longitude.Use the kmeans++ for increasing the accuracy of kmeans. But still result does not change much.\\\\nHere is my db index graph for kmeans++\\\\n![For Kmeans++][1]\\\\n\\\\n\\\\nFor kmeans\\\\n\\\\n![kmeans][2]\\\\n\\\\n\\\\nI could not decide which should be the optimal number of clusters.\\\\nI also plot the sum of squares of clusters.\\\\nFor kmeans++\\\\n\\\\n![kmeans++][3]\\\\n\\\\nfor kmeans\\\\n\\\\n![kmeans][4]\\\\n\\\\nWhat should I do? Give me suggestion.\\\\n\\\\n\\\\n [1]: http://i.stack.imgur.com/2QD1Z.png\\\\n [2]: http://i.stack.imgur.com/iNlnX.png\\\\n [3]: http://i.stack.imgur.com/10Ojw.png\\\\n [4]: http://i.stack.imgur.com/qHgik.png"], ["Give version of Mathematica", "Added the number of obs"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I clustered data according to latitude and longitude.Use the kmeans++ for increasing the accuracy of kmeans. But still result does not change much.\nHere is my db index graph for kmeans++\n![\"For](\"http://i.stack.imgur.com/2QD1Z.png\")
For kmeans
\n\n![\"kmeans\"](\"http://i.stack.imgur.com/iNlnX.png\")
I could not decide which should be the optimal number of clusters.\nI also plot the sum of squares of clusters.\nFor kmeans++
\n\n![\"kmeans++\"](\"http://i.stack.imgur.com/10Ojw.png\")
for kmeans
\n\n![\"kmeans\"](\"http://i.stack.imgur.com/qHgik.png\")
What should I do? Give me suggestion.
\n\nHere is the value of boot and noise using clusterboot\n![\"boot](\"http://i.stack.imgur.com/vpj5K.png\")
I'm trying to mine that Stack Exchange data dump to find out whether there is a cluster of users that may be positively or negatively affected by the number of badges they've been awarded. The theory I'm working on is that for some people who were motivated by badges, that they eventually lose their appeal and they stop posting.
\n\nCurrently, I've filtered the raw data down to a set of:
\n\n- \n
- User ID \n
- Year and Month \n
- Posts made during that month \n
- Total badges that user has earned up to that month \n
I'm not sure how to proceed in order to find these users. I'm not even sure how to formulate this question into something I can take action on. If I visualize it in my head, what I'm thinking is seeing a chart of an x-axis of badges earned, a y-axis of posts made per month, and then seeing the line dip after a certain number of badges. But this only works on a per user level. I want to find the cluster of users for whom that chart would hold.
\n\nAny ideas?
\n\nThank you!
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["How can people emulate a RNG; Give the 'most random' sequence of numbers", "How does the number of parameters scale with the number of features?"], ["A designer learning to code. Give me a wide birth. :)
\n", "Donate a cup of food for free: Click to Give @ The Hunger Site
\n"], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "What is the title for the post which got the highest score comment?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1, 92240], [2, 55746], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1, 221292], [1, 115376], [90, 57], ["What is the \"conclusion\" to which the title refers?", "@Alexis - I do not know the pairing of which pre score goes with which post score"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["This is much healthier.", "a_kats"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1, 386848], [38, 37], [1, 115378], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["Players of a certain TRPG have characters with 6 ability scores, each ability score ranging from 3-18. One method of generating those is by rolling 4d6 drop lowest. That means four six-faced-dice are rolled, and the three highest results are added. \\\\n\\\\nWhat's the probability that, given 5 players, one player will have a highest ability score equal or lower than the lowest ability score of another player? \\\\n\\\\nWhat's the expected highest difference in the sum of the ability scores between 2 of those 5 players? \\\\n\\\\nA good answer would explain the results in a way that a statistics novice can follow. There's a similar question [here](http://stats.stackexchange.com/questions/30585/repeatedly-rolling-a-six-sided-die-four-times-and-summing-the-highest-three-resu), but the answers it got cannot be reasonably expected to be understood by interested roleplayers without a statistics background.", "I am preparing to analyze some data and I have a question about how to go about it. It is a 39 question survey with 5-item Likert scale responses. I will be using the responses to separate the participants into the category that they score highest in for further analysis. The problem is that Category A corresponds to 14 questions, Category B has 11 questions, and Category C has 14 questions. What is the best way to go about determining which one the participant scores highest in, since the # of questions is not equal? Will it work to take their total possible points in each category and divide it by the number of points possible, then compare their percentages across the three categories to see which is the highest?"], ["Now we got the title right!", "clarified the main issue of the post in the title"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1, 115378], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1, 8169], ["I am sorry for the vague title - its because I don't know what I am talking about.
\nImagine a committee of people in charge of hiring a CEO. A committee member can vote \"No hire\" (+0), \"Maybe\"(+1), \"Hire\"(+2) for a potential CEO candidate. Each CEO is scored based on the votes, and the CEO with the highest score is hired.
Now, imagine a situation where two candidates are tied... both of them have a score of 5. However, candidate1 had 2Hires and 1Maybe (ie, 2+2+1), whereas candidate2 had 5Maybes (ie, 1+1+1+1+1).
\n\nHere is my question:
\n\n- \n
- Intuitively, I think we can say that we are more confident about candidate1 eventho the scores are tied. What specific domain of statistics am I dealing with here (so that I can investigate on my own) \n
- In statistics, how does one go about resolving such situations? And I am not just merely talking about resolving a tie... but rather, how to resolve this in a generic manner? \n
Thank you.
\n", "My survey is based on observations and interviews. The observation questions are yes/no type questions and interview questions are 4 points.
\n\n\n\n\n$1=$ never, $2=$ sometimes, $3=$ usually, and $4=$ always.
\n
\n Yes $=1$, No $=0$.
If I add up all these with highest possible score and divide it by the total score received, I can then retrieve a total score for the survey.
\n\nFor example, let's say I have 5 responses for the Likert and 5 for the yes/no. $5\\\\times4$ $=20$ (with 4 being the highest possible score you can get on the Likert scale)\nand $5\\\\times1$ $=5$ $(1$ being the highest possible score for yes/no question). I then add up both to find the highest possible score $=25$. If I get a combined score of $20$, do I divide it by the possible score, $20\\\\div25=80$%? Can I build my questionnaire and calculations like this?
\n\nMy main objective is to get one percentage / score for the entire questionnaire. I thought of accomplishing this in 2 ways:
\n\n- \n
- Get the sum of earned score from both scales divided by total possible scores for both scales. \n
- Get a percentage by using the above calculation for individual scales, which will give me two sets of percentages, and then take an average for those percentages. \n
I am not sure which way I should go with, or if this even makes sense doing it this way.
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["What is the intuition behind the score function?", "What is the training score used for?"], ["What do I look for in a post? Clarity and Contribution.
\n\nNO vote: If the post is clear, but makes no contribution OR if the post is unclear, but makes a contribution (though I will try to improve the clarity either by comments or direct edit).
\n\nUP vote: If the post is clear and makes a contribution.
\n\nDOWN vote: If the post is unclear and makes no contribution (though I will try to see if I've missed something by commenting).
\n", "Another PHP programmer responsible for things like the MicroMVC PHP framework.\\\\\\\\n\\\\\\\\nYou may notice that some of my questions run very deep into the heart of certain issues because I strive for the highest optimization that is possible in my apps."], [0, 4069], [0, 6054], [1920, 126], [1, 2621439], [13, 94], ["https://www.gravatar.com/avatar/c5a24c56df5d0bb88091a1e2869ced40?s=128&d=identicon&r=PG&f=1", "http://i.stack.imgur.com/kzMPP.jpg?s=128&g=1"]], "column_description": ["the user id", "the user's reputation", "the creation date of the user account", "the user's display name", "the last access date of the user account", "the website url of the user account", "user's location", "the self introduction of the user", "the number of views ", "the number of upvotes", "the number of downvotes", "the unique id of the account", "user's age", "the profile image url"], "value_description": ["", "", "Creation Date", "Display Name", "Last Access Date", "Website Url", "", "About Me", "", "", "", "Account Id", "", "Profile Image Url"]}, {"table_name_original": "votes", "table_name": "votes", "column_names": ["id", "post id", "vote type id", "creation date", "user id", "bounty amount"], "column_names_original": ["id", "postid", "votetypeid", "creationdate", "userid", "bountyamount"], "column_types": ["integer", "integer", "integer", "date", "integer", "integer"], "db_contents": [[1, 43538], [1, 16921], [16, 15], ["2011-02-14", "2010-08-07"], [2050, 2044], [25, 150]], "column_description": ["the vote id", "the id of the post that is voted", "the id of the vote type", "the creation date of the vote", "the id of the voter", "the amount of bounty"], "value_description": ["", "Post Id", "Vote Type Id", "Creation Date", "User Id", "Bounty Amount"]}], "pk": [{"table_name_original": "badges", "column_name_original": "id"}, {"table_name_original": "comments", "column_name_original": "id"}, {"table_name_original": "posthistory", "column_name_original": "id"}, {"table_name_original": "postlinks", "column_name_original": "id"}, {"table_name_original": "posts", "column_name_original": "id"}, {"table_name_original": "tags", "column_name_original": "id"}, {"table_name_original": "users", "column_name_original": "id"}, {"table_name_original": "votes", "column_name_original": "id"}], "fk": [{"source_table_name_original": "badges", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "comments", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posthistory", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "relatedpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "postlinks", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "parentid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "owneruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "posts", "source_column_name_original": "lasteditoruserid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "tags", "source_column_name_original": "excerptpostid", "target_table_name_original": "posts", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "userid", "target_table_name_original": "users", "target_column_name_original": "id"}, {"source_table_name_original": "votes", "source_column_name_original": "postid", "target_table_name_original": "posts", "target_column_name_original": "id"}], "table_labels": [0, 0, 0, 0, 0, 0, 0, 0], "column_labels": [[0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]}, {"question": "For the post which got 1910 view counts, how many comments does it get?", "db_id": "codebase_community", "sql": "", "norm_sql": "", "sql_skeleton": "", "natsql": "", "norm_natsql": "", "natsql_skeleton": "", "db_schema": [{"table_name_original": "badges", "table_name": "badges", "column_names": ["id", "user id", "name", "date"], "column_names_original": ["id", "userid", "name", "date"], "column_types": ["integer", "integer", "text", "datetime"], "db_contents": [[1910, 30382], [1910, 14151], ["Socratic", "Notable Question"], ["2011-04-14 01:26:03.0", "2014-05-01 15:27:06.0"]], "column_description": ["the badge id", "the unique id of the user", "the badge name the user obtained", "the date that the user obtained the badge"], "value_description": ["", "User Id", "", ""]}, {"table_name_original": "comments", "table_name": "comments", "column_names": ["id", "post id", "score", "text", "creation date", "user id", "user display name"], "column_names_original": ["id", "postid", "score", "text", "creationdate", "userid", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text"], "db_contents": [[1910, 67584], [1, 115376], [90, 57], ["Can you post the error which you get? How does MAP choke?", "How does a higher Y value \"explain\" a bigger chunk of the variance? Is it how the PCA is computed? If so I think I've got another question to post ;)"], ["2014-06-20 05:13:27.0", "2013-06-06 10:14:05.0"], [3, 45571], ["user30366", "user3487"]], "column_description": ["the comment Id", "the unique id of the post", "rating score", "the detailed content of the comment", "the creation date of the comment", "the id of the user who post the comment", "user's display name"], "value_description": ["", "Post Id", "", "", "Creation Date", "User Id", "User Display Name"]}, {"table_name_original": "posthistory", "table_name": "posthistory", "column_names": ["id", "post history type id", "post id", "revision guid", "creation date", "user id", "text", "comment", "user display name"], "column_names_original": ["id", "posthistorytypeid", "postid", "revisionguid", "creationdate", "userid", "text", "comment", "userdisplayname"], "column_types": ["integer", "integer", "integer", "text", "datetime", "integer", "text", "text", "text"], "db_contents": [[1910, 118983], [38, 37], [1910, 34289], ["c55144ab-f178-4f10-8b35-76d5b846c6f7", "3e59d7c5-ee62-4907-93f4-88138ee3ef3e"], ["2011-10-05 08:12:59.0", "2013-10-24 02:21:22.0"], [3, -1], ["I have the data of how many users post how many questions. For example, \\\\n [UserCount, QuestionCount] \\\\n [2, 100] \\\\n [9, 10] \\\\n [3, 80] \\\\n ... ...\\\\n\\\\nit means each of the 2 users posts 100 questions, each of the 9 users posts 10 questions, and so on. So, how to validate whether the UserCount-QuestionCount distribution follows a power law?\\\\n\\\\nI find the poweRlaw packages, e.g., http://cran.r-project.org/web/packages/poweRlaw/vignettes/examples.pdf. However, i can only pass one group of numbers to do the evaluation.(The provided exmaple of this package is word frequency.) So how do i use this package? Or i got something wrong? I also have the data of each user's question count, i.e., [100, 100, 10, 10, 10 ... ]. If i pass this data to the package, what i will get?", "Currently, I am doing a clustering analysis for two sets of data. One smaller dataset (about 100 data) got ground truth labels, and one larger dataset (about 2000 data) has no ground truth labels.\\\\n\\\\nFor the smaller dataset, obviously, I can obtain quantitative results like accuracy, sensitivity and specificity.\\\\n\\\\nHowever, for the larger dataset, I have no ground truth and couldn't get any useful quantitative results.\\\\n\\\\n1. The only thing I found useful is the 'mean silhouette value', which can measure the cluster performance. However, it based on some distance measure that can only tell people how separate are the clusters. I am wondering if there are other 'better' or 'more appropriate' quantitative analysis for data without labels.\\\\n\\\\n2. Because the data are without labels, I am also wondering if we can somehow have a 'uncertainty' measure about the clustering results like how confident about the cluster results?\\\\n\\\\n3. For the smaller dataset with labels, except accuracy, sensitivity and specificity, any other quantitative results I can get? For the classification algorithm, we can do a cross-validation, is there any method we can use to do such a cross-validation for clustering? Also, can we get ROC analysis for clustering task?\\\\n\\\\nThanks!\\\\nA.\\\\n "], ["cran task view does not answer this question", "For the last comment"], ["user38406", "user3487"]], "column_description": ["the post history id", "the id of the post history type", "the unique id of the post", "the revision globally unique id of the post", "the creation date of the post", "the user who post the post", "the detailed content of the post", "comments of the post", "user's display name"], "value_description": ["", "Post History Type Id", "Post Id", "Revision GUID", "Creation Date", "User Id", "", "", "User Display Name"]}, {"table_name_original": "postlinks", "table_name": "postlinks", "column_names": ["id", "creation date", "post id", "related post id", "link type id"], "column_names_original": ["id", "creationdate", "postid", "relatedpostid", "linktypeid"], "column_types": ["integer", "datetime", "integer", "integer", "integer"], "db_contents": [[2916354, 2523134], ["2013-09-19 07:42:18.0", "2012-07-27 15:36:24.0"], [65537, 32767], [1, 65523], [3, 1]], "column_description": ["the post link id", "the creation date of the post link", "the post id", "the id of the related post", "the id of the link type"], "value_description": ["", "Creation Date", "Post Id", "Related Post Id", "Link Type Id"]}, {"table_name_original": "posts", "table_name": "posts", "column_names": ["id", "post type id", "accepted answer id", "creation date", "score", "view count", "body", "owner user id", "last activity date", "title", "tags", "answer count", "comment count", "favorite count", "last editor user id", "last edit date", "community owned date", "parentid", "closed date", "owner display name", "last editor display name"], "column_names_original": ["id", "posttypeid", "acceptedanswerid", "creaiondate", "score", "viewcount", "body", "owneruserid", "lasactivitydate", "title", "tags", "answercount", "commentcount", "favoritecount", "lasteditoruserid", "lasteditdate", "communityowneddate", "parentid", "closeddate", "ownerdisplayname", "lasteditordisplayname"], "column_types": ["integer", "integer", "integer", "datetime", "integer", "integer", "text", "integer", "datetime", "text", "text", "integer", "integer", "integer", "integer", "datetime", "datetime", "integer", "datetime", "text", "text"], "db_contents": [[1910, 34283], [7, 6], [32768, 65533], ["2011-10-05 08:12:59.0", "2014-03-17 08:16:11.0"], [0, -10], [1910, 1257], ["
When one bootstraps a parameter to get the standard error we get a distribution of the parameter. Why don't we use the mean of that distribution as a result or estimate for the parameter we are trying to get? Shouldn't the distribution approximate the real one? Therefore we would get a good estimate of the \"real\" value? Yet we report the original parameter we got from our sample. Why is that?
\n\nThanks
\n", "Currently, I am doing a clustering analysis for two sets of data. One smaller dataset (about 100 data) got ground truth labels, and one larger dataset (about 2000 data) has no ground truth labels.
\n\nFor the smaller dataset, obviously, I can obtain quantitative results like accuracy, sensitivity and specificity.
\n\nHowever, for the larger dataset, I have no ground truth and couldn't get any useful quantitative results.
\n\n- \n
The only thing I found useful is the 'mean silhouette value', which can measure the cluster performance. However, it based on some distance measure that can only tell people how separate are the clusters. I am wondering if there are other 'better' or 'more appropriate' quantitative analysis for data without labels.
\nBecause the data are without labels, I am also wondering if we can somehow have a 'uncertainty' measure about the clustering results like how confident about the cluster results?
\nFor the smaller dataset with labels, except accuracy, sensitivity and specificity, any other quantitative results I can get? For the classification algorithm, we can do a cross-validation, is there any method we can use to do such a cross-validation for clustering? Also, can we get ROC analysis for clustering task?
\n
Thanks!\nA.
\n"], [5, -1], ["2013-09-26 13:09:10.0", "2011-08-11 18:53:48.0"], ["For which distributions does uncorrelatedness imply independence?", "Chi-square test for equality of distributions: how many zeroes does it tolerate?"], ["