Lecture Summarization 02

Test_text-summarizer/README.md

+### Understand Text Summarization and create your own summarizer in python
+
+Summarization can be defined as a task of producing a concise and fluent summary while preserving key information and overall meaning.
+
+Impact
+Summarization systems often have additional evidence they can utilize in order to specify the most important topics of document(s). For example, when summarizing blogs, there are discussions or comments coming after the blog post that are good sources of information to determine which parts of the blog are critical and interesting.
+
+In scientific paper summarization, there is a considerable amount of information such as cited papers and conference information which can be leveraged to identify important sentences in the original paper.
+
+Complete article with code and results can be found on [Medium](https://towardsdatascience.com/understand-text-summarization-and-create-your-own-summarizer-in-python-b26a9f09fc70)

Test_text-summarizer/java.txt

+Java is a general-purpose programming language that is class-based, object-oriented, and designed to have as few implementation dependencies as possible. It is intended to let application developers write once, run anywhere (WORA),meaning that compiled Java code can run on all platforms that support Java without the need for recompilation.Java applications are typically compiled to bytecode that can run on any Java virtual machine (JVM) regardless of the underlying computer architecture. The syntax of Java is similar to C and C++, but it has fewer low-level facilities than either of them.
\ No newline at end of file

Test_text-summarizer/msft.txt

+In an attempt to build an AI-ready workforce, Microsoft announced Intelligent Cloud Hub which has been launched to empower the next generation of students with AI-ready skills. Envisioned as a three-year collaborative program, Intelligent Cloud Hub will support around 100 institutions with AI infrastructure, course content and curriculum, developer support, development tools and give students access to cloud and AI services. As part of the program, the Redmond giant which wants to expand its reach and is planning to build a strong developer ecosystem in India with the program will set up the core AI infrastructure and IoT Hub for the selected campuses. The company will provide AI development tools and Azure AI services such as Microsoft Cognitive Services, Bot Services and Azure Machine Learning.According to Manish Prakash, Country General Manager-PS, Health and Education, Microsoft India, said, "With AI being the defining technology of our time, it is transforming lives and industry and the jobs of tomorrow will require a different skillset. This will require more collaborations and training and working with AI. That’s why it has become more critical than ever for educational institutions to integrate new cloud and AI technologies. The program is an attempt to ramp up the institutional set-up and build capabilities among the educators to educate the workforce of tomorrow." The program aims to build up the cognitive skills and in-depth understanding of developing intelligent cloud connected solutions for applications across industry. Earlier in April this year, the company announced Microsoft Professional Program In AI as a learning track open to the public. The program was developed to provide job ready skills to programmers who wanted to hone their skills in AI and data science with a series of online courses which featured hands-on labs and expert instructors as well. This program also included developer-focused AI school that provided a bunch of assets to help build AI skills.
\ No newline at end of file

Test_text-summarizer/text-summarizer.py

+import nltk
+from nltk.corpus import stopwords
+from nltk.cluster.util import cosine_distance
+import numpy as np
+import networkx as nx
+ 
+def read_article(file_name):
+    file = open(file_name, "r")
+    filedata = file.readlines()
+    article = filedata[0].split(". ")
+    sentences = []
+
+    for sentence in article:
+        print(sentence)
+        sentences.append(sentence.replace("[^a-zA-Z]", " ").split(" "))
+    sentences.pop() 
+    
+    return sentences
+
+def sentence_similarity(sent1, sent2, stopwords=None):
+    if stopwords is None:
+        stopwords = []
+ 
+    sent1 = [w.lower() for w in sent1]
+    sent2 = [w.lower() for w in sent2]
+ 
+    all_words = list(set(sent1 + sent2))
+ 
+    vector1 = [0] * len(all_words)
+    vector2 = [0] * len(all_words)
+ 
+    # build the vector for the first sentence
+    for w in sent1:
+        if w in stopwords:
+            continue
+        vector1[all_words.index(w)] += 1
+ 
+    # build the vector for the second sentence
+    for w in sent2:
+        if w in stopwords:
+            continue
+        vector2[all_words.index(w)] += 1
+ 
+    return 1 - cosine_distance(vector1, vector2)
+ 
+def build_similarity_matrix(sentences, stop_words):
+    # Create an empty similarity matrix
+    similarity_matrix = np.zeros((len(sentences), len(sentences)))
+ 
+    for idx1 in range(len(sentences)):
+        for idx2 in range(len(sentences)):
+            if idx1 == idx2: #ignore if both are same sentences
+                continue 
+            similarity_matrix[idx1][idx2] = sentence_similarity(sentences[idx1], sentences[idx2], stop_words)
+
+    return similarity_matrix
+
+
+def generate_summary(file_name, top_n=5):
+    nltk.download("stopwords")
+    stop_words = stopwords.words('english')
+    summarize_text = []
+
+    # Step 1 - Read text anc split it
+    sentences =  read_article(file_name)
+
+    # Step 2 - Generate Similary Martix across sentences
+    sentence_similarity_martix = build_similarity_matrix(sentences, stop_words)
+
+    # Step 3 - Rank sentences in similarity martix
+    sentence_similarity_graph = nx.from_numpy_array(sentence_similarity_martix)
+    scores = nx.pagerank(sentence_similarity_graph)
+
+    # Step 4 - Sort the rank and pick top sentences
+    ranked_sentence = sorted(((scores[i],s) for i,s in enumerate(sentences)), reverse=True)    
+    print("Indexes of top ranked_sentence order are ", ranked_sentence)    
+
+    for i in range(top_n):
+      summarize_text.append(" ".join(ranked_sentence[i][1]))
+
+    # Step 5 - Offcourse, output the summarize texr
+    print("Summarize Text: \n", ". ".join(summarize_text))
+
+# let's begin
+generate_summary( "msft.txt", 2)